The Principia Machina/TPC-PY/tpc falcon.py: Difference between revisions
TPC-PY source tpc_falcon.py (0.2.0), SHA-256 ae5775ff27683078; conformed TPS 0.4.0; PROPOSED, not declared (TPC-PY 0.2.0) |
m 1 revision imported |
(No difference)
| |
Latest revision as of 02:11, 30 September 2026
"""tpc_falcon 0.2.0 (DRAFT, pending D148, D149, D150, D153, D162 and D164;
0.1.1: the SERVED source kind of D172 (c), taken by delegation 2026-09-26,
tpm_biblio_resolution 0.1.0 RD-17 and 6.14; 0.2.0, proposed MINOR with
TPC-PY 0.2.0: the output-schema digest TPS 5.20.2 requires, S79, agent
RDC's specification FS1-FS5): the falcon draw, the falcon reconciliation
with the QUOTE CHECK's pure parts and the source match, the per-kind stop,
the sponsor-stage predicate, and the digest of a FALCON KIND's output
schema.
Reference implementation, Python standard library only. DATA for the
Falcon round (tpm_falcon_round_pause 0.1.0, the authority; the falcon
speculation 0.1.0 sections 6-8; the onboarding speculation 0.1.0 sections
5-7); not a declared TPC-PY version and not TPX code. Every function is a
pure function of its arguments. Every refused argument raises ValueError
with a short fixed message. render_*() give the canonical text that the
PHP residence (tpc_falcon.php) must reproduce byte for byte. Objects are
JSON objects (dict), lists are JSON arrays (list); ints are never bools.
falcon_draw(candidates, n, state, r) -> dict (D149 Option 1)
candidates: list of objects, one per agent-group account, with exactly
account, sponsor, family (D156's VENDOR FAMILY) and snapshot (model
snapshot): non-empty strings; tier (TRUST TIER), good_standing (GOOD
STANDING, D126), sandbox (scoped to the sandboxes, D124) and over_cap
(at its falcon cap, D155 (e)): booleans; window (falcons in the last
FALCON WINDOW), ontime and assigned (FALCON STRATUM counts, D153):
integers, 0 <= ontime <= assigned. Accounts unique.
n: the new assignees wanted, 1 to 3 (2 at issue, 3 in a FALCON TRIAL,
1 for D150's L2 third assignee or for a reissue).
state: kept (accounts whose assignments stand and will be compared with
the new ones; each in candidates), prior (every earlier assignee of
this falcon, whether it returned or not, kept ones included: D149 (b);
never drawn again), drafted and now
(integer seconds, UTC), falcon_wait (FALCON WAIT, seconds).
len(kept) + n >= 2: never fewer than two agents (D149 (a)).
r: the supplied random value, 32 lowercase hexadecimal characters.
Rule:
DW1 ELIGIBLE (b): good_standing, not sandbox, not over_cap, and no
earlier assignee of the falcon (kept or prior). So one account is
never drawn twice (a).
DW2 ORDER (g): window ascending; then the on-time share ontime/assigned
descending, compared exactly by cross-multiplication (an account with
no assignment counts as share 1); then h ascending, h = SHA-256 over
the D97 netstring of ["TPC-FALCON-DRAW", "1", r, account]; then
account.
DW3 NEW PAIRS: the pairs among the new accounts and between each new
account and each kept one. The rungs and the marks read these only.
DW4 RUNGS (c), the first that holds:
T1 no new pair shares a sponsor; every new account in the tier;
T2 no new pair shares a sponsor; some new or kept account in the tier;
T3 no new pair shares a sponsor;
T4 every new pair that shares a sponsor is two tier accounts (the
user's top-tier relaxation);
T5 any set, and only under THIN POOL.
With no kept account and n = 2 these are D149 (c)'s T1-T5 word for
word; with kept accounts they are D149 (h)'s "same rule". T3 comes
before T4: D149 Option 4 (T4 before T3) is a lower option and is not
implemented. For a first pair T4 is reached only when the eligible
accounts hold one sponsor, so THIN POOL holds too (D149's EFFECT).
DW5 THIN POOL (f), counted: fewer than two distinct sponsors among the
eligible accounts; or now - drafted >= falcon_wait and no set of n
eligible accounts satisfies T3.
DW6 INSIDE THE BEST RUNG (d, e, g, h): fewest new pairs sharing a
sponsor ("if possible", which matters on T4 and T5 only when three
accounts meet); then fewest new pairs of one family; then fewest of
one snapshot; then the set whose members stand first in ORDER
(lexicographic over their sorted positions). With kept accounts the
family key is (h)'s "a third vendor family first".
DW7 RESULT: status OK, accounts (in ORDER), rung, marks over the new
pairs (SAME-SPONSOR, SAME-VENDOR, SAME-MODEL), thin, eligible and
sponsors; or status SHORT (fewer than n eligible accounts) or WAIT (no
set on T1-T4 and no THIN POOL yet). Under SHORT and WAIT the falcon
stays DRAFTED: one assignee is never drawn alone (the exclusion).
falcon_canon(value, spec) -> str (D150 (b))
spec.type, the canonical forms and nothing beyond them (a false DIFFER
costs a rung; a false AGREED is the soundness defect):
text a string; NFC, then XML Schema's whiteSpace "collapse" (TAB,
LF and CR become SPACE, runs of SPACE become one, leading and
trailing SPACE removed). No other character is changed.
decimal a string, the zone serialization's DECIMAL (zone schemas
5.6): -?digits(.digits)?([eE][-+]?d{1,4})?; "0" for zero,
else [-]M with "e"E when E is not 0, M an integer without
leading or trailing zeros. String arithmetic only.
unitvalue a string "<decimal> <UNIT>", split at the first SPACE; the
decimal as above and the UNIT in its canonical spelling
(tpm_units 0.2.0 sections 2-5: the closed catalogue, the 24
prefixes, one reading per token, merge, exponent and
DIMENSION range -9999..9999, catalogue order then prefix
power). "5 cm" and "0.05 m" stay different (D104.1).
date a string, an ISO 8601 calendar date YYYY-MM-DD or YYYYMMDD,
a real day of 0000-9999; canonical YYYY-MM-DD.
url a string of RFC 3986 characters with a scheme; only the
syntax-based steps of RFC 3986 6.2.2: scheme and host to
lower case, percent-encoding hex to upper case, unreserved
characters decoded, dot segments removed (5.2.4) from a path
under an authority or beginning with "/". A default port,
an empty port, path case and query order are kept (6.2.3 is
scheme-based and not taken).
enum a string, one of spec.values, as it is.
boolean true or false, as "true" or "false".
Anything else raises ValueError (the result is then MALFORMED).
falcon_reconcile(outputs, quotes, schema) -> dict (D150 Option 1)
schema: class (FINDINGS, PATCH or SURVEY: D148 (b)), narrow (this falcon
is the narrower reissue of SINGLE rows: D150 (g)), confidential
(optional boolean: the kind is declared CONFIDENTIAL, D148 (d), D152
(f)), fields: a list of field specs {name, type, class (MECHANICAL or
INSPECTION), quote (optional boolean: the field carries a quote, D150
(c)), values (enum only), affix and scale (optional, on a decimal or
unitvalue field with quote true only: its QUOTE AFFIX and QUOTE SCALE,
each a string falcon_quote_match accepts; TPX passes them to
quote_check for that field's values)}; type "rows" adds key (column
names) and columns (field specs, no rows), its key columns
MECHANICAL. Names are ASCII identifiers, unique; "IV" is reserved.
Anything else raises ("schema").
outputs: a list of at least two {account, status (RETURNED or OVERDUE),
result}; result null is a refusal; accounts distinct.
quotes: a list parallel to outputs of objects item id -> PASS, FAIL or
UNKNOWN: TPX's QUOTE CHECK outcome for that output's value (quote_check
over the page text TPX holds, under the field's affix and scale and
the source entry's accounting flag; UNKNOWN also for a citation
outside every declared entry, or a wiki page the packet did not
serve); a missing entry is UNKNOWN.
Item ids: a field's name; a keyed row's "<name>[<key>]" and its non-key
column's "<name>[<key>].<column>", <key> the D97 netstring of the key
columns' canonical values.
Rule:
RL1 VALIDATE (FM-5, D150 (h)): an OVERDUE output, a null result (a
refusal) or a result that is not an object with exactly the schema's
field names (plus an optional IV list of strings, never compared)
whose every value passes falcon_canon is not usable. Reason: OVERDUE,
"refusal", "shape", "IV" or the first failing field. Rows: a list of
objects with exactly the columns, canonical key unique. If any output
is not usable the disposition is REISSUE (those indices) and nothing
is compared: MALFORMED is never a disagreement.
RL2 ITEMS: fields in schema order; a rows field expands to its rows in
key order. A row carried by one output only is SINGLE; otherwise
each non-key column is an item voted on by the outputs carrying the
row.
RL3 L0 (D150 (b), (i)): an INSPECTION item is INSPECTION, never matched.
A MECHANICAL item whose voters' canonical values are all equal is
AGREED; otherwise it is SPLIT.
RL4 L1 (D150 (c)): on a SPLIT item whose spec carries a quote, if exactly
one distinct value has a PASS among its voters, the item is settled
QUOTE with that value.
RL5 L2 (D150 (d), (e)): an item still SPLIT while the falcon has two
outputs waits for a third assignee (disposition THIRD). With three
or more outputs a value held by a strict majority of the item's
voters settles it MAJORITY.
RL6 L3 (D150 (f)): otherwise UNRESOLVED, shown to the approver inside
the one approval form with every value.
RL7 DISPOSITION: REISSUE (RL1); THIRD (any item SPLIT); NARROW (SINGLE
rows, schema.narrow false and the kind not CONFIDENTIAL: TPX issues
the narrower falcon of those rows once, D150 (g), and writes
nothing, no FALCON PAGE and no value in the FALCON LOG, until it
returns, D150 (a); it then publishes this falcon's result with the
narrower one's); else PRIVATE REPORT for a CONFIDENTIAL FINDINGS
kind (D152 (f): its findings go only through TPX to the restricted
page), REPORTED for FINDINGS (D151 (f)) and AWAITING APPROVAL for
PATCH and SURVEY (D151). The single field: "narrow" under NARROW,
"mark" (the rows reach the approver or the report marked SINGLE)
when schema.narrow is true or the kind is CONFIDENTIAL (whose SINGLE
rows are never reissued, D150 (g)), "wait" under THIRD, "-" when
none. The blind third review of INSPECTION prose (D150 (i)) is
TPX's step after this function.
falcon_quote_match(value, quote_text, affix=None, scale=None,
accounting=False) -> PASS or FAIL (D150 (c), TOKEN RULE)
quote_check(page_text, quote, value, affix=None, scale=None,
accounting=False) -> PASS, FAIL or UNKNOWN
affix: the field's QUOTE AFFIX or null: a word of TK6's two lists
(percent, per cent, pct; per mille, permille, bp, bps, basis point,
basis points, pp, percentage point, percentage points, ppm, ppb, ppt,
ppq), or 1 to 4 characters, the first a TK6 symbol and any others
letters ("%", "$", U+00B0 "C"). scale: the field's QUOTE SCALE or
null: any entry of TK7's catalogue as listed there (a word in the
singular). accounting: the page's FALCON SOURCE LIST entry is declared
to write -5 as "(5)". Anything else raises ("affix", "scale",
"accounting"; "text" for a text that is not a string).
TK0 CHARACTERS. Every character of Unicode category Cf (U+200B,
U+00AD, U+2060, U+FEFF and the rest) is removed from each text, the
value included; each is then put in canon_text's form (NFC,
whitespace collapsed); no case folding. Categories come from the
class table embedded in both residences, generated by
gen_unicode_table.py from Python's unicodedata at Unicode 15.1.0
and checked against its SHA-256 on first use: L (Lu Ll Lt Lm Lo), M
(Mn Mc Me), D (Nd), N (Nl No), P (Pd), S (Sm), C (Sc), Z (Zs), F
(Cf). Below, a "digit" is D or N (superscript and fraction digits
count) and a "space" is a run of Z characters (U+0020, U+00A0,
U+2002, U+2007, U+2009, U+202F and U+3000 among them).
TK1 WHOLE TOKEN. An occurrence of the needle counts only where it
neither begins nor ends inside a word or a number: the character
before it and after it is not L, M, D or N, and neither boundary is
joined into a number token by TK2-TK7. Any occurrence that passes
is enough; an empty needle FAILS.
TK2 SIGN (T1). A P or S character ("+", "-", U+2212, U+2013, U+00B1,
"<", ">", "=", "~", U+2248, U+2264, U+2265 among them) joins what
follows it when that is a digit, a "." or "," before a digit, or a
sign or prefix symbol already joined to one (a run); across a space
only when the sign itself follows no digit, spaces between: "5"
FAILS in "-5", "+5", "<=5", "~5", "-$5", "- 5" and "x = 5".
TK3 SEPARATORS (T2). ".", ",", "'", U+2019 or "_" between two digits;
a "." or "," immediately before a digit; a space between a digit
and a group of exactly three digits (followed by no further
digit): "5" FAILS in "1,5", ".5", "5.5"; "234" in "1,234", "1 234"
and "1" U+2007 "234".
TK4 JOINS (T3). Between two digits, touching both or with a space on
either side: every P (Pd) character of the table (the hyphen-minus,
U+2010-U+2015, U+2E3A, U+2E3B, U+301C, U+FE58, U+FE63, U+FF0D and
the rest of Pd), and U+2212, "/", ":", "~", U+FF5E, U+2044, U+2215,
U+2236, U+00F7, U+29F8 and U+2053: "3" and "5" FAIL in "3-5" and
"3 - 5", "5" in "5/10", "5 / 10", "5" U+2044 "10", "5" U+2014
"10", "5 " U+2014 " 10", "5" U+00F7 "10", "5~10" and "5" U+2053
"10", "2" in "2:1" and "2" U+2236 "1". MIXED NUMBERS: after the
last digit of a number, a space and then a fraction join: a vulgar
fraction (U+00BC-U+00BE, U+2150-U+215F, U+2189), or a digit run,
then "/", U+2044 or U+2215, then a digit: "5" FAILS in "5 "
U+00BD, "5 1/2" and "5" U+00A0 "1" U+2044 "2" ("5" U+00BD joins by
TK1).
TK5 EXPONENTS (T4). "e" or "E" after a digit and before a sign and a
digit ("e" itself is a letter); "^" after a digit, with the digit or
signed digit after it; U+207A or U+207B after a digit; after the
last digit of a number, a multiplication sign ("*", U+00D7, U+00B7,
or "x" or "X" with a space on each side), spaces allowed, then "10"
not followed by a decimal digit: "1" FAILS in "1e-5", "1 x 10^-5"
and "1" U+00D7 "10" U+207B U+2075; "10" in "10^5".
TK6 AFFIXES (T5). Before or after a number, touching it or across a
space: "%", U+2030, U+2031, U+00B0, U+2032, U+2033, U+FF05, U+FE6A,
U+066A, U+0609, U+060A or any Sc character; the whole word
"percent", "pct" or "per cent". After a number only, touching it or
across a space: the whole word per mille, permille, bp, bps, basis
point, basis points, pp, percentage point, percentage points, ppm,
ppb, ppt or ppq. Words are ASCII in any case; a two-word form takes
any space between its words. "5" FAILS in "5%", "5 %", "5" U+2002
"%", "5" U+FF05, "5 per cent", "5 PCT", "percent 5", "$5", "5 bps",
"5 basis points", "5 pp", "5 percentage points", "5 ppm", "5 PPB"
and "5 per mille".
TK7 SCALE (T8). After the last digit of a number, touching it or across
a space: the whole word hundred, thousand, million, billion,
trillion, quadrillion, lakh, crore, dozen, tenth, hundredth,
thousandth, millionth, billionth or trillionth, or its plural in
"s" (ASCII, any case); the whole abbreviation mn, bn, mln, bln, tn,
trn, mio or mrd (ASCII, any case); the whole abbreviation k, K, M,
m, B or MM (exact case; a bare "m" is metres or million and joins
likewise); "e" or "E" alone, then an optional space, an optional
sign and a digit (a spaced exponent); "**", then an optional space,
an optional sign and a digit; or the word "times" and a space
before "10^": "5" FAILS in "5 million", "5 Millions", "5 M", "5 k",
"5 bn", "5 Bn", "5 mln", "5 BLN", "5 tn", "5 trn", "5 Mio", "5 MM",
"5 dozen", "5 thousandths", "5 m", "$5 billion", "5 E-3", "5 e 3"
and "5 times 10^3"; "10" in "10**5". "5 mm", "5 km", "5 models" and
"12 metres" join nothing (TK10).
TK8 DECLARED FORMS. A needle ending in a digit may end at a suffix
affix or scale, and one beginning with a digit, sign or leading
point may begin at a prefix affix (a symbol, "percent", "pct" or
"per cent"), only when the declared affix or scale is what stands
there (touching or across a space; a word or a case-free
abbreviation in any ASCII case, a scale word also as its plural, a
declared two-word form ending in "point" or "points" with either
number, a symbol or an exact-case abbreviation exactly) and the
token ends beyond it. A suffix may carry the scale and the affix,
each once, in either order. With "%" declared "72.4" PASSES in
"72.4%" and "72.4 %"; with "million" "5" PASSES in "5 million";
with "$" and "billion" in "$5 billion"; with "basis point" in "5
basis points". An exponent or power product (TK5, TK7) is never
declarable, so a value inside one always FAILS.
TK9 THE VALUE'S OWN FORM (T6). The needle is matched as written, sign
included: "-5" matches "-5" only, never "+5", "5" or U+2212 "5"; no
separator is converted ("1234" FAILS against "1,234", "1.5" against
"1,5"). A miss here is a false FAIL, never a false PASS.
TK10 OUTSIDE THE TOKEN (T7). Brackets, parentheses, quotation marks, a
"." or "," not followed by a digit, and a word that is neither an
affix nor a scale join nothing: "12" PASSES in "(12)", "is 12." and
"12 metres"; "5" in "5-shot". With accounting true, each "("
number ")" (ASCII digits with TK3 separators and spaces) is read as
"-" number in every text first: "5" FAILS and "-5" PASSES in "(5)".
THE NAMED LIMIT: the check claims exactly the catalogues of T1-T8
(TK2-TK7). A scale or affix written in a word outside them, a
range or fraction in words ("5 to 10", "5 and a half") and a unit
whose prefix scales ("5 km" quoted for a field in metres) pass.
The remaining guards are the second agent's independent value at
L2 and the approver, who reads every quote on the approval form.
TK11 quote_check: UNKNOWN when page_text is null (no page TPX may read);
PASS when the quote is a whole token of the page text and the value
a whole token of the quote, each under the same affix, scale and
accounting; else FAIL. The value is the agent's text, not its
canonical DECIMAL.
source_match(cited_url, source_list) -> dict (D150 (c), FM-15)
source_hop(hop_url, matched, source_list) -> dict
source_list: the FALCON SOURCE LIST, a list of entries [kind, url] with
kind EXACT or PREFIX, and of CODE HOST marks ["CODE HOST", host,
template, ...]; from 0.1.1 also SERVED entries ["SERVED", url], one
per registry record TPX fetched under a REGISTRY DECLARATION and
served in the packet, url the registry_url it fetched (D172 (c),
RD-17), which TPX adds to a falcon packet's list. Each url in
canon_url's form must be https with a host name (no userinfo, no IP
literal, no host of digits and dots, no "%"), a port only as ":" and
digits, no fragment, a path as written that SM2 does not refuse, and
no query on a PREFIX; else it raises ("sources"). A CODE HOST mark names a lower-case host name once; each
template is a path of segments ending in "/", "*" standing for any
one non-empty segment ("/*/*/tree/*/": organisation, repository,
"tree", branch).
SM1 PUBLISHER TEST, at the list's validation (D150 (c)): a PREFIX whose
path is "/" (a host root) raises ("publisher"); so does a PREFIX on
a host the list marks CODE HOST unless its path reaches, segment by
segment, at least as deep as one of the host's templates, so an
organisation or repository root, and anything above a named branch
or commit, is refused; a CODE HOST with no template refuses every
PREFIX on it. A template's last "*" is its branch or commit slot; a
PREFIX whose segment there is "refs" in any ASCII case does not
reach that template, since refs expose pull-request refs, anyone's
content (N2). Whether only the publisher writes beneath any other
prefix is H-DECL's INSPECTION, recorded with the declaration.
SM2 PATH RULES. The path as written is percent-decoded repeatedly until
it no longer changes; it is refused if any stage holds an encoded
"/" (%2F), an encoded backslash (%5C) or a backslash, or bytes that
are not valid UTF-8 (RFC 3629: an overlong form such as %C0%AE, a
surrogate, a truncated sequence; N4), if seven decodings do not
settle it, or if the settled path holds a "." or
".." segment or a ";" parameter: "/org/..;/x", "/%2E%2E/x" and the
double-encoded "/%252E%252E/x" are all refused before canon_url
could remove or keep them.
SM3 The citation is put in canon_url's form; a citation that fails, or
has no authority, is UNKNOWN "url". Then, in this order, it is
UNKNOWN "scheme" unless https, "userinfo" with userinfo, "host" for
an IP literal, a host of digits and dots or a "%" in the host, and
"path" when SM2 refuses its path.
SM4 An empty path is "/" (the request target it fetches).
SM5 An entry is a candidate only with the citation's host and port
exactly (a port other than the declared one, or an explicit :443
against none, matches nothing).
SM6 EXACT: the paths are equal and the entry names no query or the same
query. PREFIX: the path equals the prefix, or begins with it at a
path-segment boundary (the prefix ends in "/" or the next character
is "/"), so "/a/b" never covers "/a/bc". An EXACT entry wins; else
a SERVED entry (SM10); else the longest prefix; the first in list
order among equals.
SM7 The citation's query is dropped unless the entry names it; its
fragment is always dropped.
SM8 Result: ENTRY with the entry's index, kind and declared URL, and the
URL TPX fetches: the declared URL for EXACT; for PREFIX the URL
rebuilt from the entry's https, host and port and the citation's
canonical path, which may equal the citation: that is safe only
because the whole prefix is the publisher's (SM1). No candidate:
UNKNOWN "none". TC-20's other guards (a declared host, pinned
resolution) and the wiki-page rule are TPX's.
SM9 source_hop: each redirect hop, resolved by TPX to an absolute URL,
is matched again at entry level by SM2-SM8, not host level.
matched is the index of the EXACT or PREFIX entry the citation
matched (anything else, a SERVED index included, raises "matched":
a SERVED entry is read from the store and never fetched, so no hop
leaves it). UNKNOWN ends the fetch as UNKNOWN (render "hop|END"); a
hop inside an entry is "hop|SAME" when that is the matched entry
and "hop|OTHER" otherwise, and the fetch goes on under the entry
the hop matched. A SERVED entry is no candidate for a hop: the
store is not on the network, so a hop onto a registry URL is
UNKNOWN "none" unless an EXACT or PREFIX entry covers it.
SM10 SERVED (0.1.1; D172 (c), RD-17): a citation equal to a SERVED
entry after canon_url, the canonical paths equal and the queries
equal (both absent or the same; the fragment dropped by SM7), is
ENTRY kind SERVED with "fetch": "STORE": TPX reads the record from
its store as served, no network. The SERVED URL is validated as an
EXACT URL is except for SM2's path rules and the PUBLISHER TEST:
nothing is fetched, so an encoded ";" or "<" that registry_url put
in the path (RD-15) is no hazard, and a citation whose path SM2
refuses still matches a SERVED entry it equals (the test is exact
canonical equality with a URL TPX itself built and fetched under
THE FETCH's guards). Render "source|ENTRY|<i>|SERVED|<url>" then
"fetch|STORE".
falcon_stop(counts, limits) -> dict (D148 (g), D153 (d), FM-14)
counts: the kind's records in recorded order: ["ISSUED", id];
["FINISHED", id, items, declined, unresolved], the falcon closed with
its item count and its DECLINED and UNRESOLVED items (each item
counted once; a falcon closed without items gives 0, 0, 0);
["RESUMED"], H-OP's act.
limits: breaker (FALCON BREAKER, a decimal string from "0" to "1", at
most four places) and window (the last 20 falcons; 1 to 1000).
SP1 A falcon's share is (declined + unresolved) / items over the last
window falcons finished since the kind's declaration or its latest
resumption; it "passes" when strictly above breaker (exact integer
arithmetic); no items, no share.
SP2 TPX stops issuing the kind at the first FINISHED after which the
share passes; it stays stopped until RESUMED.
SP3 On RESUMED the count restarts, and one falcon issues alone first:
only when none of the kind is in flight, and no other is issued
until it finishes; the kind stops again if the share (now that
falcon's) passes. A falcon issued before the resumption does not
count after it.
SP4 RESUMED while issuing changes nothing.
SP5 Result: ISSUING when TPX may issue a falcon of the kind now, else
STOPPED; with the share's bad and total items and falcons counted,
the falcon issuing alone (waiting, its id, or none) and the number
in flight. The vocabulary is D148 (g)'s: stops, resumes.
SP6 A record TPX could not have made raises ("counts"): an ISSUED while
STOPPED or beside the falcon issuing alone, an id issued twice, a
FINISHED for a falcon not in flight, declined + unresolved above
items.
sponsor_stage(events, as_of, thresholds) -> dict (D162, D164 Option 1)
events: a list of [kind, time] or [kind, time, agent], time integer
seconds UTC in 0..253402300799, agent a non-empty string:
LANDED agent a work block landed by one of the sponsor's agents,
under any certificate, probes included (D162 (c))
EVALUATED a W1 weekly evaluation of the predicate
BLOCKED, UNBLOCKED the sponsor's block set or lifted
SUSPENDED, RESTORED agent an agent's SUSPENDED state set or cleared
VIOLATION agent a confirmed violation by an agent (never cleared)
as_of: integer seconds UTC; events after it are not read.
thresholds: blocks (B), dates (D), span_days (S) and dormant_days
(T_idle), integers; SETS holds D162's sets A, B and C.
Rule:
ST1 Events are read in time order, input order among equal times; an
evaluation at time t sees every event at or before t.
ST2 The predicate is evaluated at every LANDED and EVALUATED time (D162:
"at each landing ... and at W1's weekly run"): G1 at least B landed
blocks; G2 on at least D distinct UTC dates (time // 86400); G3 at
least S days from the first landed block to the latest; G4 the
sponsor not blocked, no agent SUSPENDED, no VIOLATION to date.
ST3 Stage: SERIOUS if G1-G4 held at any evaluation (one-way, D162 (f));
else WORKING if a block has landed; else NEW. Only counted events
decide; no score, rank or statistic.
ST4 DORMANT (D164 Option 1): a block has landed and as_of minus the
latest landing is at least dormant_days days; a display qualifier
that lowers no stage.
falcon_schema_canon(schema) -> ["OK", text] or ["REFUSED", "schema"]
falcon_schema_digest(schema) -> ["OK", digest] or ["REFUSED", "schema"]
(0.2.0; TPS 5.20.2, D148 (d): the digest of the output schema a FALCON
KIND DECLARATION states; S79; agent RDC's FS1-FS5, after S49's
module_digest and D171's entry digest.)
FS1 INPUT: a schema falcon_reconcile's schema check accepts, every
string of which, keys included, is already its own canonical text:
it holds no CR and is in NFC (FS specification 0.2.0; D213.1, taken
2026-09-28); anything else is refused, reason "schema". A string
holding a lone surrogate is refused likewise (it has no UTF-8;
TPC-PHP cannot decode one).
FS2 NORMAL FORM: "narrow" is removed (TPX's flag for a narrower
reissue, not a property of the kind); every "quote": false and a
"confidential": false are removed (absent means false), so two
spellings of one meaning digest alike.
FS3 CANONICAL TEXT: JSON with no white space outside strings; each
object's members in ascending code point order of their keys; arrays
in their given order (field, column, key and enumeration order carry
meaning); each string, keys included, in tpc_zone's canonical text
(CRLF and CR to LF, then NFC) between quotation marks, the quotation
mark and the backslash escaped by a backslash, each code point below
U+0020 as \\u00xx (lowercase hex), every other code point as itself,
in UTF-8; true and false as such. A schema holds no number.
FS4 DIGEST: SHA-256 over the D97 netstring of ["TPC-FALCON-SCHEMA", "1",
the canonical text], 64 lowercase hexadecimal characters.
FS5 RENDERS: "canon|" and the text as it is (no render escape: the text
holds no control character and is the last field); "digest|" and the
hex; "ERROR:schema".
FS1's canonical-text rule (D213.1) makes FS3's line-end and NFC steps
the identity on every schema accepted, so equal digests mean equal
strings: two schemas whose enumeration values or QUOTE AFFIXes differ
only in normal form or line ends no longer digest alike, while
falcon_canon goes on matching enumeration values exactly (D150 (b);
collision C-TP-4 of tpm_increment_11 0.2.0, closed).
"""
import bisect
import hashlib
import re
import unicodedata
from functools import cmp_to_key
VERSION = "0.2.0" # proposed MINOR with TPC-PY 0.2.0: the output-schema digest (S79)
DAY = 86400
TIME_MAX = 253402300799
COUNT_MAX = 1000000000
SETS = {
"A": {"blocks": 10, "dates": 5, "span_days": 14, "dormant_days": 90},
"B": {"blocks": 5, "dates": 3, "span_days": 7, "dormant_days": 90},
"C": {"blocks": 50, "dates": 10, "span_days": 30, "dormant_days": 90},
}
# ---------------------------------------------------------------- common
def _esc(s):
"""Render escape: "|" and "\\" backslashed; C0 controls and DEL as
\\xHH; every other character as it is."""
out = []
for ch in s:
o = ord(ch)
if ch == "|" or ch == "\\":
out.append("\\" + ch)
elif o < 0x20 or o == 0x7F:
out.append("\\x%02x" % o)
else:
out.append(ch)
return "".join(out)
def _netstring(fields):
return b"".join(b"%d:%s" % (len(b), b)
for b in (f.encode("utf-8") for f in fields))
def _digest_fields(fields):
return hashlib.sha256(_netstring(fields)).hexdigest()
def _is_int(x, lo, hi):
return type(x) is int and lo <= x <= hi
def _is_name(x):
return type(x) is str and x != ""
def _keys_are(o, required, optional=()):
if type(o) is not dict:
return False
for k in required:
if k not in o:
return False
for k in o:
if k not in required and k not in optional:
return False
return True
# ---------------------------------------------------------- canonical forms
_WS = re.compile(r"[\t\n\r ]+")
_DEC = re.compile(r"(-?)([0-9]+)(?:\.([0-9]+))?(?:[eE]([-+]?[0-9]{1,4}))?")
_TOK = re.compile(r"([A-Za-z]+)(-?[0-9]{1,4})?")
_DATE1 = re.compile(r"([0-9]{4})-([0-9]{2})-([0-9]{2})")
_DATE2 = re.compile(r"([0-9]{4})([0-9]{2})([0-9]{2})")
_URLCH = re.compile(r"[A-Za-z0-9\-._~:/?#\[\]@!$&'()*+,;=%]+")
_BADPCT = re.compile(r"%(?![0-9A-Fa-f]{2})")
_PCT = re.compile(r"%([0-9A-Fa-f]{2})")
_APPB = re.compile(r"(?:([^:/?#]+):)?(?://([^/?#]*))?([^?#]*)"
r"(?:\?([^#]*))?(?:#(.*))?")
_SCHEME = re.compile(r"[A-Za-z][A-Za-z0-9+.\-]*")
_USERINFO = re.compile(r"[A-Za-z0-9\-._~!$&'()*+,;=:%]*")
_REGNAME = re.compile(r"[A-Za-z0-9\-._~!$&'()*+,;=%]*")
_IPLIT = re.compile(r"\[[A-Za-z0-9\-._~!$&'()*+,;=:%]*\]")
_PORT = re.compile(r"(?::[0-9]*)?")
_HOSTCASE = re.compile(r"%[0-9A-F]{2}|[A-Z]+")
_UNRESERVED = ("ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz"
"0123456789-._~")
# tpm_units 0.2.0 section 3: the closed catalogue in canonical order, with
# DIMENSION exponents over L M T I TH N J (section 6).
_CATALOGUE = [
("m", (1, 0, 0, 0, 0, 0, 0)), ("g", (0, 1, 0, 0, 0, 0, 0)),
("s", (0, 0, 1, 0, 0, 0, 0)), ("A", (0, 0, 0, 1, 0, 0, 0)),
("K", (0, 0, 0, 0, 1, 0, 0)), ("mol", (0, 0, 0, 0, 0, 1, 0)),
("cd", (0, 0, 0, 0, 0, 0, 1)), ("rad", (0, 0, 0, 0, 0, 0, 0)),
("sr", (0, 0, 0, 0, 0, 0, 0)), ("Hz", (0, 0, -1, 0, 0, 0, 0)),
("N", (1, 1, -2, 0, 0, 0, 0)), ("Pa", (-1, 1, -2, 0, 0, 0, 0)),
("J", (2, 1, -2, 0, 0, 0, 0)), ("W", (2, 1, -3, 0, 0, 0, 0)),
("C", (0, 0, 1, 1, 0, 0, 0)), ("V", (2, 1, -3, -1, 0, 0, 0)),
("F", (-2, -1, 4, 2, 0, 0, 0)), ("Ohm", (2, 1, -3, -2, 0, 0, 0)),
("S", (-2, -1, 3, 2, 0, 0, 0)), ("Wb", (2, 1, -2, -1, 0, 0, 0)),
("T", (0, 1, -2, -1, 0, 0, 0)), ("H", (2, 1, -2, -2, 0, 0, 0)),
("lm", (0, 0, 0, 0, 0, 0, 1)), ("lx", (-2, 0, 0, 0, 0, 0, 1)),
("Bq", (0, 0, -1, 0, 0, 0, 0)), ("Gy", (2, 0, -2, 0, 0, 0, 0)),
("Sv", (2, 0, -2, 0, 0, 0, 0)), ("kat", (0, 0, -1, 0, 0, 1, 0)),
("min", (0, 0, 1, 0, 0, 0, 0)), ("h", (0, 0, 1, 0, 0, 0, 0)),
("d", (0, 0, 1, 0, 0, 0, 0)), ("au", (1, 0, 0, 0, 0, 0, 0)),
("deg", (0, 0, 0, 0, 0, 0, 0)), ("arcmin", (0, 0, 0, 0, 0, 0, 0)),
("arcsec", (0, 0, 0, 0, 0, 0, 0)), ("ha", (2, 0, 0, 0, 0, 0, 0)),
("L", (3, 0, 0, 0, 0, 0, 0)), ("t", (0, 1, 0, 0, 0, 0, 0)),
("eV", (2, 1, -2, 0, 0, 0, 0)),
]
_CATIDX = {s: i for i, (s, _) in enumerate(_CATALOGUE)}
_NOPREFIX = ("min", "h", "d", "ha", "au", "deg", "arcmin")
_REFUSED = ("degC", "Da", "Np", "B", "dB")
_PREFIXES = [("q", -30), ("r", -27), ("y", -24), ("z", -21), ("a", -18),
("f", -15), ("p", -12), ("n", -9), ("u", -6), ("m", -3),
("c", -2), ("d", -1), ("da", 1), ("h", 2), ("k", 3), ("M", 6),
("G", 9), ("T", 12), ("P", 15), ("E", 18), ("Z", 21),
("Y", 24), ("R", 27), ("Q", 30)]
_PREFIX_OF = {p: s for s, p in _PREFIXES}
TYPES = ("text", "decimal", "unitvalue", "date", "url", "enum", "boolean")
def canon_text(s):
if type(s) is not str:
raise ValueError("text")
for ch in s:
if 0xD800 <= ord(ch) <= 0xDFFF:
raise ValueError("text")
s = unicodedata.normalize("NFC", s)
return _WS.sub(" ", s).strip(" ")
def canon_decimal(s):
if type(s) is not str:
raise ValueError("decimal")
m = _DEC.fullmatch(s)
if m is None:
raise ValueError("decimal")
sign = m.group(1)
frac = m.group(3) if m.group(3) is not None else ""
e = int(m.group(4)) if m.group(4) is not None else 0
e -= len(frac)
digits = (m.group(2) + frac).lstrip("0")
if digits == "":
return "0"
trimmed = digits.rstrip("0")
e += len(digits) - len(trimmed)
return sign + trimmed + ("e%d" % e if e != 0 else "")
def _unit_token(w):
"""(catalogue index, prefix power) of one token's letters."""
if w in _CATIDX:
return (_CATIDX[w], 0)
if w in _REFUSED:
raise ValueError("unit")
found = []
for pre, pw in _PREFIXES:
if len(w) > len(pre) and w[:len(pre)] == pre:
rest = w[len(pre):]
if rest in _REFUSED:
raise ValueError("unit")
if rest in _CATIDX and rest not in _NOPREFIX:
found.append((_CATIDX[rest], pw))
if len(found) != 1:
raise ValueError("unit")
return found[0]
def canon_unit(u):
if type(u) is not str:
raise ValueError("unit")
if u == "1":
return "1"
merged = {}
for part in u.split(" "):
m = _TOK.fullmatch(part)
if m is None:
raise ValueError("unit")
key = _unit_token(m.group(1))
ex = int(m.group(2)) if m.group(2) is not None else 1
merged[key] = merged.get(key, 0) + ex
dim = [0, 0, 0, 0, 0, 0, 0]
keep = []
for key in sorted(merged):
ex = merged[key]
if ex < -9999 or ex > 9999:
raise ValueError("unit")
if ex == 0:
continue
sd = _CATALOGUE[key[0]][1]
for j in range(7):
dim[j] += sd[j] * ex
keep.append((key, ex))
for x in dim:
if x < -9999 or x > 9999:
raise ValueError("unit")
if not keep:
return "1"
parts = []
for (ci, pw), ex in keep:
t = (_PREFIX_OF[pw] if pw != 0 else "") + _CATALOGUE[ci][0]
parts.append(t if ex == 1 else t + str(ex))
return " ".join(parts)
def canon_date(s):
if type(s) is not str:
raise ValueError("date")
m = _DATE1.fullmatch(s)
if m is None:
m = _DATE2.fullmatch(s)
if m is None:
raise ValueError("date")
y, mo, d = int(m.group(1)), int(m.group(2)), int(m.group(3))
leap = y % 4 == 0 and (y % 100 != 0 or y % 400 == 0)
if mo < 1 or mo > 12:
raise ValueError("date")
last = [31, 29 if leap else 28, 31, 30, 31, 30, 31, 31, 30, 31, 30,
31][mo - 1]
if d < 1 or d > last:
raise ValueError("date")
return "%04d-%02d-%02d" % (y, mo, d)
def _pct(m):
c = chr(int(m.group(1), 16))
if c in _UNRESERVED:
return c
return "%" + m.group(1).upper()
def _host_case(m):
t = m.group(0)
return t if t[0] == "%" else t.lower()
def remove_dot_segments(path):
"""RFC 3986 5.2.4, step by step."""
inp = path
out = ""
while inp != "":
if inp[:3] == "../":
inp = inp[3:]
elif inp[:2] == "./":
inp = inp[2:]
elif inp[:3] == "/./":
inp = "/" + inp[3:]
elif inp == "/.":
inp = "/"
elif inp[:4] == "/../":
inp = "/" + inp[4:]
k = out.rfind("/")
out = out[:k] if k >= 0 else ""
elif inp == "/..":
inp = "/"
k = out.rfind("/")
out = out[:k] if k >= 0 else ""
elif inp == "." or inp == "..":
inp = ""
else:
k = inp.find("/", 1) if inp[:1] == "/" else inp.find("/")
if k < 0:
out += inp
inp = ""
else:
out += inp[:k]
inp = inp[k:]
return out
def canon_url(s):
if type(s) is not str or _URLCH.fullmatch(s) is None:
raise ValueError("url")
if _BADPCT.search(s) is not None:
raise ValueError("url")
s = _PCT.sub(_pct, s)
m = _APPB.fullmatch(s)
scheme, auth, path, query, frag = m.groups()
if scheme is None or _SCHEME.fullmatch(scheme) is None:
raise ValueError("url")
out = scheme.lower() + ":"
if auth is not None:
if auth.count("@") > 1:
raise ValueError("url")
if "@" in auth:
ui, hp = auth.split("@", 1)
if _USERINFO.fullmatch(ui) is None:
raise ValueError("url")
ui += "@"
else:
ui, hp = "", auth
if hp[:1] == "[":
k = hp.find("]")
if k < 0:
raise ValueError("url")
host, port = hp[:k + 1], hp[k + 1:]
if _IPLIT.fullmatch(host) is None:
raise ValueError("url")
else:
k = hp.find(":")
host, port = (hp, "") if k < 0 else (hp[:k], hp[k:])
if _REGNAME.fullmatch(host) is None:
raise ValueError("url")
if _PORT.fullmatch(port) is None:
raise ValueError("url")
out += "//" + ui + _HOSTCASE.sub(_host_case, host) + port
if auth is not None or path[:1] == "/":
path = remove_dot_segments(path)
out += path
if query is not None:
out += "?" + query
if frag is not None:
out += "#" + frag
return out
def _check_spec(spec, allow_rows):
"""Validates one field spec; raises ValueError("schema")."""
if type(spec) is not dict or type(spec.get("type")) is not str:
raise ValueError("schema")
t = spec["type"]
base = ("name", "type", "class")
if t == "enum":
ok = _keys_are(spec, base + ("values",), ("quote",))
elif t == "rows" and allow_rows:
ok = _keys_are(spec, base + ("key", "columns"))
elif t in ("decimal", "unitvalue"):
ok = _keys_are(spec, base, ("quote", "affix", "scale"))
elif t in TYPES:
ok = _keys_are(spec, base, ("quote",))
else:
ok = False
if not ok:
raise ValueError("schema")
nm = spec["name"]
if type(nm) is not str or re.fullmatch(r"[A-Za-z][A-Za-z0-9_]*",
nm) is None or nm == "IV":
raise ValueError("schema")
if spec["class"] not in ("MECHANICAL", "INSPECTION"):
raise ValueError("schema")
if "quote" in spec and type(spec["quote"]) is not bool:
raise ValueError("schema")
for k, check in (("affix", _check_affix), ("scale", _check_scale)):
if k in spec:
if spec.get("quote") is not True or type(spec[k]) is not str:
raise ValueError("schema")
try:
check(spec[k])
except ValueError:
raise ValueError("schema")
if t == "enum":
v = spec["values"]
if type(v) is not list or not v:
raise ValueError("schema")
for i, x in enumerate(v):
if type(x) is not str or x in v[:i]:
raise ValueError("schema")
if t == "rows":
if spec["class"] != "MECHANICAL":
raise ValueError("schema")
cols = spec["columns"]
if type(cols) is not list or not cols:
raise ValueError("schema")
names = []
for c in cols:
_check_spec(c, False)
if c["name"] in names:
raise ValueError("schema")
names.append(c["name"])
key = spec["key"]
if type(key) is not list or not key:
raise ValueError("schema")
for i, k in enumerate(key):
if k not in names or k in key[:i]:
raise ValueError("schema")
if cols[names.index(k)]["class"] != "MECHANICAL":
raise ValueError("schema")
def falcon_canon(value, spec):
_check_spec(spec, False)
t = spec["type"]
if t == "text":
return canon_text(value)
if t == "decimal":
return canon_decimal(value)
if t == "unitvalue":
if type(value) is not str or " " not in value:
raise ValueError("unitvalue")
k = value.find(" ")
try:
return canon_decimal(value[:k]) + " " + canon_unit(value[k + 1:])
except ValueError:
raise ValueError("unitvalue")
if t == "date":
return canon_date(value)
if t == "url":
return canon_url(value)
if t == "enum":
if type(value) is not str or value not in spec["values"]:
raise ValueError("enum")
return value
if type(value) is not bool:
raise ValueError("boolean")
return "true" if value else "false"
def render_canon(c):
return "OK|" + _esc(c)
# ------------------------------------------------------------------- draw
_CKEYS = ("account", "sponsor", "family", "snapshot", "tier",
"good_standing", "sandbox", "over_cap", "window", "ontime",
"assigned")
_SKEYS = ("kept", "prior", "drafted", "now", "falcon_wait")
def _draw_check(candidates, n, state, r):
if type(candidates) is not list:
raise ValueError("candidates")
seen = set()
for c in candidates:
if not _keys_are(c, _CKEYS):
raise ValueError("candidates")
for k in ("account", "sponsor", "family", "snapshot"):
if not _is_name(c[k]):
raise ValueError("candidates")
for k in ("tier", "good_standing", "sandbox", "over_cap"):
if type(c[k]) is not bool:
raise ValueError("candidates")
for k in ("window", "ontime", "assigned"):
if not _is_int(c[k], 0, COUNT_MAX):
raise ValueError("candidates")
if c["ontime"] > c["assigned"] or c["account"] in seen:
raise ValueError("candidates")
seen.add(c["account"])
if not _is_int(n, 1, 3):
raise ValueError("n")
if not _keys_are(state, _SKEYS):
raise ValueError("state")
for k in ("kept", "prior"):
if type(state[k]) is not list:
raise ValueError("state")
for i, a in enumerate(state[k]):
if not _is_name(a) or a in state[k][:i]:
raise ValueError("state")
for a in state["kept"]:
if a not in seen:
raise ValueError("state")
for k in ("drafted", "now", "falcon_wait"):
if not _is_int(state[k], 0, TIME_MAX):
raise ValueError("state")
if state["now"] < state["drafted"]:
raise ValueError("state")
if len(state["kept"]) + n < 2:
raise ValueError("n")
if type(r) is not str or re.fullmatch(r"[0-9a-f]{32}", r) is None:
raise ValueError("r")
def _order_cmp(a, b):
if a["window"] != b["window"]:
return -1 if a["window"] < b["window"] else 1
na, da = (a["ontime"], a["assigned"]) if a["assigned"] > 0 else (1, 1)
nb, db = (b["ontime"], b["assigned"]) if b["assigned"] > 0 else (1, 1)
x, y = na * db, nb * da
if x != y:
return -1 if x > y else 1
if a["_h"] != b["_h"]:
return -1 if a["_h"] < b["_h"] else 1
if a["account"] != b["account"]:
return -1 if a["account"] < b["account"] else 1
return 0
def _combos(m, n):
"""Index sets of size n (1 to 3) in lexicographic order."""
if n == 1:
return [(i,) for i in range(m)]
if n == 2:
return [(i, j) for i in range(m) for j in range(i + 1, m)]
return [(i, j, k) for i in range(m) for j in range(i + 1, m)
for k in range(j + 1, m)]
def falcon_draw(candidates, n, state, r):
_draw_check(candidates, n, state, r)
by = {c["account"]: c for c in candidates}
kept = [by[a] for a in state["kept"]]
excluded = set(state["kept"]) | set(state["prior"])
elig = []
for c in candidates:
if (c["good_standing"] and not c["sandbox"] and not c["over_cap"]
and c["account"] not in excluded):
d = dict(c)
d["_h"] = _digest_fields(["TPC-FALCON-DRAW", "1", r,
c["account"]])
elig.append(d)
order = sorted(elig, key=cmp_to_key(_order_cmp))
sponsors = len({c["sponsor"] for c in order})
waited = state["now"] - state["drafted"] >= state["falcon_wait"]
res = {"eligible": len(order), "sponsors": sponsors}
if len(order) < n:
res.update(status="SHORT", thin=sponsors < 2 or waited)
return res
kept_tier = any(k["tier"] for k in kept)
best = None
any_t3 = False
for idxs in _combos(len(order), n):
s = [order[i] for i in idxs]
pairs = [(s[i], s[j]) for i in range(n) for j in range(i + 1, n)]
pairs += [(x, k) for x in s for k in kept]
shared = [p for p in pairs if p[0]["sponsor"] == p[1]["sponsor"]]
if not shared:
any_t3 = True
if all(x["tier"] for x in s):
rung = 0
elif kept_tier or any(x["tier"] for x in s):
rung = 1
else:
rung = 2
elif all(p[0]["tier"] and p[1]["tier"] for p in shared):
rung = 3
else:
rung = 4
fam = sum(1 for p in pairs if p[0]["family"] == p[1]["family"])
snap = sum(1 for p in pairs if p[0]["snapshot"] == p[1]["snapshot"])
key = (rung, len(shared), fam, snap)
if best is None or key < best[0]:
best = (key, idxs)
thin = sponsors < 2 or (waited and not any_t3)
res["thin"] = thin
(rung, nshared, fam, snap), idxs = best
if rung == 4 and not thin:
res["status"] = "WAIT"
return res
marks = []
if nshared:
marks.append("SAME-SPONSOR")
if fam:
marks.append("SAME-VENDOR")
if snap:
marks.append("SAME-MODEL")
res.update(status="OK", rung="T%d" % (rung + 1), marks=marks,
accounts=[order[i]["account"] for i in idxs])
return res
def render_draw(d):
lines = ["draw|" + d["status"] + ("|" + d["rung"] if d["status"] == "OK"
else "")]
if d["status"] == "OK":
for a in d["accounts"]:
lines.append("assignee|" + _esc(a))
lines.append("marks|" + (",".join(d["marks"]) or "-"))
lines.append("thin|" + ("true" if d["thin"] else "false"))
lines.append("pool|%d|%d" % (d["eligible"], d["sponsors"]))
return "\n".join(lines)
# -------------------------------------------------------------- reconcile
def _check_schema(schema):
if not _keys_are(schema, ("class", "narrow", "fields"),
("confidential",)):
raise ValueError("schema")
if schema["class"] not in ("FINDINGS", "PATCH", "SURVEY"):
raise ValueError("schema")
if type(schema["narrow"]) is not bool:
raise ValueError("schema")
if type(schema.get("confidential", False)) is not bool:
raise ValueError("schema")
fields = schema["fields"]
if type(fields) is not list or not fields:
raise ValueError("schema")
names = []
for f in fields:
_check_spec(f, True)
if f["name"] in names:
raise ValueError("schema")
names.append(f["name"])
def _validate(out, schema):
"""(reason, values, iv): reason None when usable."""
if out["status"] == "OVERDUE":
return ("OVERDUE", None, 0)
res = out["result"]
if res is None:
return ("refusal", None, 0)
names = [f["name"] for f in schema["fields"]]
if not _keys_are(res, names, ("IV",)):
return ("shape", None, 0)
iv = res.get("IV", [])
if type(iv) is not list:
return ("IV", None, 0)
for x in iv:
if type(x) is not str:
return ("IV", None, 0)
vals = {}
for f in schema["fields"]:
v = res[f["name"]]
try:
if f["type"] != "rows":
vals[f["name"]] = falcon_canon(v, f)
continue
if type(v) is not list:
raise ValueError("rows")
cnames = [c["name"] for c in f["columns"]]
rows = {}
for row in v:
if not _keys_are(row, cnames):
raise ValueError("rows")
cv = {}
for c in f["columns"]:
cv[c["name"]] = falcon_canon(row[c["name"]], c)
kenc = _netstring([cv[k] for k in f["key"]]).decode("utf-8")
if kenc in rows:
raise ValueError("rows")
rows[kenc] = cv
vals[f["name"]] = rows
except ValueError:
return (f["name"], None, 0)
return (None, vals, len(iv))
def falcon_reconcile(outputs, quotes, schema):
_check_schema(schema)
if type(outputs) is not list or len(outputs) < 2:
raise ValueError("outputs")
seen = []
for o in outputs:
if not _keys_are(o, ("account", "status", "result")):
raise ValueError("outputs")
if not _is_name(o["account"]) or o["account"] in seen:
raise ValueError("outputs")
seen.append(o["account"])
if o["status"] not in ("RETURNED", "OVERDUE"):
raise ValueError("outputs")
if o["status"] == "OVERDUE" and o["result"] is not None:
raise ValueError("outputs")
if type(quotes) is not list or len(quotes) != len(outputs):
raise ValueError("quotes")
for q in quotes:
if type(q) is not dict:
raise ValueError("quotes")
for v in q.values():
if v not in ("PASS", "FAIL", "UNKNOWN"):
raise ValueError("quotes")
checked = [_validate(o, schema) for o in outputs]
res = {"outputs": [(o["account"], c[0], c[2])
for o, c in zip(outputs, checked)]}
bad = [i for i, c in enumerate(checked) if c[0] is not None]
if bad:
res["disposition"] = "REISSUE"
res["reissue"] = bad
return res
vals = [c[1] for c in checked]
nout = len(outputs)
items = []
for f in schema["fields"]:
nm = f["name"]
if f["type"] != "rows":
items.append((nm, f, [(i, vals[i][nm]) for i in range(nout)]))
continue
allkeys = set()
for i in range(nout):
allkeys.update(vals[i][nm].keys())
for kenc in sorted(allkeys):
rid = nm + "[" + kenc + "]"
carriers = [i for i in range(nout) if kenc in vals[i][nm]]
if len(carriers) == 1:
items.append((rid, None, carriers[0]))
continue
for c in f["columns"]:
if c["name"] in f["key"]:
continue
items.append((rid + "." + c["name"], c,
[(i, vals[i][nm][kenc][c["name"]])
for i in carriers]))
out = []
for iid, spec, voters in items:
if spec is None:
out.append((iid, "SINGLE", None, str(voters), []))
continue
quoted = spec.get("quote", False)
vq = [(i, v, quotes[i].get(iid, "UNKNOWN") if quoted else "-")
for i, v in voters]
detail = str(len(voters))
if spec["class"] == "INSPECTION":
out.append((iid, "INSPECTION", None, detail, vq))
continue
counts = {}
for i, v in voters:
counts[v] = counts.get(v, 0) + 1
if len(counts) == 1:
out.append((iid, "AGREED", voters[0][1], detail, []))
continue
if quoted:
passers = sorted({v for i, v, q in vq if q == "PASS"})
if len(passers) == 1:
out.append((iid, "QUOTE", passers[0], detail, vq))
continue
if nout == 2:
out.append((iid, "SPLIT", None, detail, vq))
continue
win = [v for v in sorted(counts) if 2 * counts[v] > len(voters)]
if win:
out.append((iid, "MAJORITY", win[0],
"%d/%d" % (counts[win[0]], len(voters)), vq))
else:
out.append((iid, "UNRESOLVED", None, detail, vq))
res["items"] = out
kinds = [x[1] for x in out]
conf = schema.get("confidential", False)
if "SPLIT" in kinds:
disp = "THIRD"
elif "SINGLE" in kinds and not schema["narrow"] and not conf:
disp = "NARROW"
elif schema["class"] == "FINDINGS":
disp = "PRIVATE REPORT" if conf else "REPORTED"
else:
disp = "AWAITING APPROVAL"
if "SINGLE" not in kinds:
single = "-"
elif disp == "THIRD":
single = "wait"
elif schema["narrow"] or conf:
single = "mark"
else:
single = "narrow"
res["disposition"] = disp
res["single"] = single
return res
def render_reconcile(d):
lines = []
for i, (acct, reason, iv) in enumerate(d["outputs"]):
if reason is None:
lines.append("output|%d|%s|OK|iv=%d" % (i, _esc(acct), iv))
elif reason == "OVERDUE":
lines.append("output|%d|%s|OVERDUE" % (i, _esc(acct)))
else:
lines.append("output|%d|%s|MALFORMED|%s" % (i, _esc(acct),
reason))
if d["disposition"] == "REISSUE":
lines.append("disposition|REISSUE|" +
",".join(str(i) for i in d["reissue"]))
return "\n".join(lines)
tally = {}
for iid, kind, value, detail, vq in d["items"]:
tally[kind] = tally.get(kind, 0) + 1
lines.append("item|%s|%s|%s|%s" % (
_esc(iid), kind, "-" if value is None else _esc(value), detail))
for i, v, q in vq:
lines.append("value|%s|%d|%s|%s" % (_esc(iid), i, _esc(v), q))
lines.append("summary|" + "|".join(
"%s=%d" % (k.lower(), tally.get(k, 0)) for k in
("AGREED", "QUOTE", "MAJORITY", "SPLIT", "UNRESOLVED", "INSPECTION",
"SINGLE")))
lines.append("disposition|%s|single=%s" % (d["disposition"],
d["single"]))
return "\n".join(lines)
# ------------------------------------------------------------------ stage
_AGENT_KINDS = ("LANDED", "SUSPENDED", "RESTORED", "VIOLATION")
_PLAIN_KINDS = ("EVALUATED", "BLOCKED", "UNBLOCKED")
def _stage_check(events, as_of, thresholds):
if type(events) is not list:
raise ValueError("events")
for e in events:
if type(e) is not list or len(e) < 2:
raise ValueError("events")
if e[0] in _AGENT_KINDS:
if len(e) != 3 or not _is_name(e[2]):
raise ValueError("events")
elif e[0] in _PLAIN_KINDS:
if len(e) != 2:
raise ValueError("events")
else:
raise ValueError("events")
if not _is_int(e[1], 0, TIME_MAX):
raise ValueError("events")
if not _is_int(as_of, 0, TIME_MAX):
raise ValueError("as_of")
if not _keys_are(thresholds, ("blocks", "dates", "span_days",
"dormant_days")):
raise ValueError("thresholds")
for k, lo in (("blocks", 1), ("dates", 1), ("span_days", 0),
("dormant_days", 1)):
if not _is_int(thresholds[k], lo, 1000000):
raise ValueError("thresholds")
def sponsor_stage(events, as_of, thresholds):
_stage_check(events, as_of, thresholds)
evs = [e for e in events if e[1] <= as_of]
evs = [e for _, e in sorted(enumerate(evs),
key=lambda p: (p[1][1], p[0]))]
b, dd = thresholds["blocks"], thresholds["dates"]
span_min = thresholds["span_days"] * DAY
first = last = None
nblocks = 0
dates = set()
blocked = False
violation = False
suspended = set()
grad = None
i = 0
while i < len(evs):
t = evs[i][1]
evaluate = False
while i < len(evs) and evs[i][1] == t:
kind = evs[i][0]
if kind == "LANDED":
nblocks += 1
dates.add(t // DAY)
if first is None:
first = t
last = t
evaluate = True
elif kind == "EVALUATED":
evaluate = True
elif kind == "BLOCKED":
blocked = True
elif kind == "UNBLOCKED":
blocked = False
elif kind == "SUSPENDED":
suspended.add(evs[i][2])
elif kind == "RESTORED":
suspended.discard(evs[i][2])
else:
violation = True
i += 1
if evaluate and grad is None and nblocks > 0:
if (nblocks >= b and len(dates) >= dd
and last - first >= span_min and not blocked
and not suspended and not violation):
grad = (t, nblocks, len(dates), last - first)
if grad is not None:
stage = "SERIOUS"
elif nblocks > 0:
stage = "WORKING"
else:
stage = "NEW"
dormant = (nblocks > 0
and as_of - last >= thresholds["dormant_days"] * DAY)
return {"stage": stage, "dormant": dormant, "blocks": nblocks,
"dates": len(dates), "span": 0 if first is None else last - first,
"last": last, "graduated": grad}
def render_stage(d):
lines = ["stage|" + d["stage"],
"dormant|" + ("true" if d["dormant"] else "false"),
"counts|%d|%d|%d|%s" % (d["blocks"], d["dates"], d["span"],
"-" if d["last"] is None
else str(d["last"]))]
g = d["graduated"]
lines.append("graduated|-" if g is None else "graduated|%d|%d|%d|%d" % g)
return "\n".join(lines)
# ---------------------------------------------------------- unicode table
# The classes the TOKEN RULE reads, generated from Python's unicodedata at a
# pinned Unicode version by gen_unicode_table.py and embedded in both
# residences: L letter, M mark, D Nd, N Nl or No, P Pd, S Sm, C Sc, Z Zs,
# F Cf; any other code point has no class.
# BEGIN UNICODE TABLE (gen_unicode_table.py; do not edit)
UNICODE_VERSION = "15.1.0"
UNICODE_TABLE_SHA256 = (
"c0bbd7cd8eb0bac9332bbf3b621efdb10538aac0852657f7e8e000ceb877d223")
_UTABLE = (
"20Z;24C;2BS;2DP;30-39D;3C-3ES;41-5AL;61-7AL;7CS;7ES;A0Z;A2-A5C;AAL;A"
"CS;ADF;B1S;B2-B3N;B5L;B9N;BAL;BC-BEN;C0-D6L;D7S;D8-F6L;F7S;F8-2C1L;2"
"C6-2D1L;2E0-2E4L;2ECL;2EEL;300-36FM;370-374L;376-377L;37A-37DL;37FL;"
"386L;388-38AL;38CL;38E-3A1L;3A3-3F5L;3F6S;3F7-481L;483-489M;48A-52FL"
";531-556L;559L;560-588L;58AP;58FC;591-5BDM;5BEP;5BFM;5C1-5C2M;5C4-5C"
"5M;5C7M;5D0-5EAL;5EF-5F2L;600-605F;606-608S;60BC;610-61AM;61CF;620-6"
"4AL;64B-65FM;660-669D;66E-66FL;670M;671-6D3L;6D5L;6D6-6DCM;6DDF;6DF-"
"6E4M;6E5-6E6L;6E7-6E8M;6EA-6EDM;6EE-6EFL;6F0-6F9D;6FA-6FCL;6FFL;70FF"
";710L;711M;712-72FL;730-74AM;74D-7A5L;7A6-7B0M;7B1L;7C0-7C9D;7CA-7EA"
"L;7EB-7F3M;7F4-7F5L;7FAL;7FDM;7FE-7FFC;800-815L;816-819M;81AL;81B-82"
"3M;824L;825-827M;828L;829-82DM;840-858L;859-85BM;860-86AL;870-887L;8"
"89-88EL;890-891F;898-89FM;8A0-8C9L;8CA-8E1M;8E2F;8E3-903M;904-939L;9"
"3A-93CM;93DL;93E-94FM;950L;951-957M;958-961L;962-963M;966-96FD;971-9"
"80L;981-983M;985-98CL;98F-990L;993-9A8L;9AA-9B0L;9B2L;9B6-9B9L;9BCM;"
"9BDL;9BE-9C4M;9C7-9C8M;9CB-9CDM;9CEL;9D7M;9DC-9DDL;9DF-9E1L;9E2-9E3M"
";9E6-9EFD;9F0-9F1L;9F2-9F3C;9F4-9F9N;9FBC;9FCL;9FEM;A01-A03M;A05-A0A"
"L;A0F-A10L;A13-A28L;A2A-A30L;A32-A33L;A35-A36L;A38-A39L;A3CM;A3E-A42"
"M;A47-A48M;A4B-A4DM;A51M;A59-A5CL;A5EL;A66-A6FD;A70-A71M;A72-A74L;A7"
"5M;A81-A83M;A85-A8DL;A8F-A91L;A93-AA8L;AAA-AB0L;AB2-AB3L;AB5-AB9L;AB"
"CM;ABDL;ABE-AC5M;AC7-AC9M;ACB-ACDM;AD0L;AE0-AE1L;AE2-AE3M;AE6-AEFD;A"
"F1C;AF9L;AFA-AFFM;B01-B03M;B05-B0CL;B0F-B10L;B13-B28L;B2A-B30L;B32-B"
"33L;B35-B39L;B3CM;B3DL;B3E-B44M;B47-B48M;B4B-B4DM;B55-B57M;B5C-B5DL;"
"B5F-B61L;B62-B63M;B66-B6FD;B71L;B72-B77N;B82M;B83L;B85-B8AL;B8E-B90L"
";B92-B95L;B99-B9AL;B9CL;B9E-B9FL;BA3-BA4L;BA8-BAAL;BAE-BB9L;BBE-BC2M"
";BC6-BC8M;BCA-BCDM;BD0L;BD7M;BE6-BEFD;BF0-BF2N;BF9C;C00-C04M;C05-C0C"
"L;C0E-C10L;C12-C28L;C2A-C39L;C3CM;C3DL;C3E-C44M;C46-C48M;C4A-C4DM;C5"
"5-C56M;C58-C5AL;C5DL;C60-C61L;C62-C63M;C66-C6FD;C78-C7EN;C80L;C81-C8"
"3M;C85-C8CL;C8E-C90L;C92-CA8L;CAA-CB3L;CB5-CB9L;CBCM;CBDL;CBE-CC4M;C"
"C6-CC8M;CCA-CCDM;CD5-CD6M;CDD-CDEL;CE0-CE1L;CE2-CE3M;CE6-CEFD;CF1-CF"
"2L;CF3M;D00-D03M;D04-D0CL;D0E-D10L;D12-D3AL;D3B-D3CM;D3DL;D3E-D44M;D"
"46-D48M;D4A-D4DM;D4EL;D54-D56L;D57M;D58-D5EN;D5F-D61L;D62-D63M;D66-D"
"6FD;D70-D78N;D7A-D7FL;D81-D83M;D85-D96L;D9A-DB1L;DB3-DBBL;DBDL;DC0-D"
"C6L;DCAM;DCF-DD4M;DD6M;DD8-DDFM;DE6-DEFD;DF2-DF3M;E01-E30L;E31M;E32-"
"E33L;E34-E3AM;E3FC;E40-E46L;E47-E4EM;E50-E59D;E81-E82L;E84L;E86-E8AL"
";E8C-EA3L;EA5L;EA7-EB0L;EB1M;EB2-EB3L;EB4-EBCM;EBDL;EC0-EC4L;EC6L;EC"
"8-ECEM;ED0-ED9D;EDC-EDFL;F00L;F18-F19M;F20-F29D;F2A-F33N;F35M;F37M;F"
"39M;F3E-F3FM;F40-F47L;F49-F6CL;F71-F84M;F86-F87M;F88-F8CL;F8D-F97M;F"
"99-FBCM;FC6M;1000-102AL;102B-103EM;103FL;1040-1049D;1050-1055L;1056-"
"1059M;105A-105DL;105E-1060M;1061L;1062-1064M;1065-1066L;1067-106DM;1"
"06E-1070L;1071-1074M;1075-1081L;1082-108DM;108EL;108FM;1090-1099D;10"
"9A-109DM;10A0-10C5L;10C7L;10CDL;10D0-10FAL;10FC-1248L;124A-124DL;125"
"0-1256L;1258L;125A-125DL;1260-1288L;128A-128DL;1290-12B0L;12B2-12B5L"
";12B8-12BEL;12C0L;12C2-12C5L;12C8-12D6L;12D8-1310L;1312-1315L;1318-1"
"35AL;135D-135FM;1369-137CN;1380-138FL;13A0-13F5L;13F8-13FDL;1400P;14"
"01-166CL;166F-167FL;1680Z;1681-169AL;16A0-16EAL;16EE-16F0N;16F1-16F8"
"L;1700-1711L;1712-1715M;171F-1731L;1732-1734M;1740-1751L;1752-1753M;"
"1760-176CL;176E-1770L;1772-1773M;1780-17B3L;17B4-17D3M;17D7L;17DBC;1"
"7DCL;17DDM;17E0-17E9D;17F0-17F9N;1806P;180B-180DM;180EF;180FM;1810-1"
"819D;1820-1878L;1880-1884L;1885-1886M;1887-18A8L;18A9M;18AAL;18B0-18"
"F5L;1900-191EL;1920-192BM;1930-193BM;1946-194FD;1950-196DL;1970-1974"
"L;1980-19ABL;19B0-19C9L;19D0-19D9D;19DAN;1A00-1A16L;1A17-1A1BM;1A20-"
"1A54L;1A55-1A5EM;1A60-1A7CM;1A7FM;1A80-1A89D;1A90-1A99D;1AA7L;1AB0-1"
"ACEM;1B00-1B04M;1B05-1B33L;1B34-1B44M;1B45-1B4CL;1B50-1B59D;1B6B-1B7"
"3M;1B80-1B82M;1B83-1BA0L;1BA1-1BADM;1BAE-1BAFL;1BB0-1BB9D;1BBA-1BE5L"
";1BE6-1BF3M;1C00-1C23L;1C24-1C37M;1C40-1C49D;1C4D-1C4FL;1C50-1C59D;1"
"C5A-1C7DL;1C80-1C88L;1C90-1CBAL;1CBD-1CBFL;1CD0-1CD2M;1CD4-1CE8M;1CE"
"9-1CECL;1CEDM;1CEE-1CF3L;1CF4M;1CF5-1CF6L;1CF7-1CF9M;1CFAL;1D00-1DBF"
"L;1DC0-1DFFM;1E00-1F15L;1F18-1F1DL;1F20-1F45L;1F48-1F4DL;1F50-1F57L;"
"1F59L;1F5BL;1F5DL;1F5F-1F7DL;1F80-1FB4L;1FB6-1FBCL;1FBEL;1FC2-1FC4L;"
"1FC6-1FCCL;1FD0-1FD3L;1FD6-1FDBL;1FE0-1FECL;1FF2-1FF4L;1FF6-1FFCL;20"
"00-200AZ;200B-200FF;2010-2015P;202A-202EF;202FZ;2044S;2052S;205FZ;20"
"60-2064F;2066-206FF;2070N;2071L;2074-2079N;207A-207CS;207FL;2080-208"
"9N;208A-208CS;2090-209CL;20A0-20C0C;20D0-20F0M;2102L;2107L;210A-2113"
"L;2115L;2118S;2119-211DL;2124L;2126L;2128L;212A-212DL;212F-2139L;213"
"C-213FL;2140-2144S;2145-2149L;214BS;214EL;2150-2182N;2183-2184L;2185"
"-2189N;2190-2194S;219A-219BS;21A0S;21A3S;21A6S;21AES;21CE-21CFS;21D2"
"S;21D4S;21F4-22FFS;2320-2321S;237CS;239B-23B3S;23DC-23E1S;2460-249BN"
";24EA-24FFN;25B7S;25C1S;25F8-25FFS;266FS;2776-2793N;27C0-27C4S;27C7-"
"27E5S;27F0-27FFS;2900-2982S;2999-29D7S;29DC-29FBS;29FE-2AFFS;2B30-2B"
"44S;2B47-2B4CS;2C00-2CE4L;2CEB-2CEEL;2CEF-2CF1M;2CF2-2CF3L;2CFDN;2D0"
"0-2D25L;2D27L;2D2DL;2D30-2D67L;2D6FL;2D7FM;2D80-2D96L;2DA0-2DA6L;2DA"
"8-2DAEL;2DB0-2DB6L;2DB8-2DBEL;2DC0-2DC6L;2DC8-2DCEL;2DD0-2DD6L;2DD8-"
"2DDEL;2DE0-2DFFM;2E17P;2E1AP;2E2FL;2E3A-2E3BP;2E40P;2E5DP;3000Z;3005"
"-3006L;3007N;301CP;3021-3029N;302A-302FM;3030P;3031-3035L;3038-303AN"
";303B-303CL;3041-3096L;3099-309AM;309D-309FL;30A0P;30A1-30FAL;30FC-3"
"0FFL;3105-312FL;3131-318EL;3192-3195N;31A0-31BFL;31F0-31FFL;3220-322"
"9N;3248-324FN;3251-325FN;3280-3289N;32B1-32BFN;3400-4DBFL;4E00-A48CL"
";A4D0-A4FDL;A500-A60CL;A610-A61FL;A620-A629D;A62A-A62BL;A640-A66EL;A"
"66F-A672M;A674-A67DM;A67F-A69DL;A69E-A69FM;A6A0-A6E5L;A6E6-A6EFN;A6F"
"0-A6F1M;A717-A71FL;A722-A788L;A78B-A7CAL;A7D0-A7D1L;A7D3L;A7D5-A7D9L"
";A7F2-A801L;A802M;A803-A805L;A806M;A807-A80AL;A80BM;A80C-A822L;A823-"
"A827M;A82CM;A830-A835N;A838C;A840-A873L;A880-A881M;A882-A8B3L;A8B4-A"
"8C5M;A8D0-A8D9D;A8E0-A8F1M;A8F2-A8F7L;A8FBL;A8FD-A8FEL;A8FFM;A900-A9"
"09D;A90A-A925L;A926-A92DM;A930-A946L;A947-A953M;A960-A97CL;A980-A983"
"M;A984-A9B2L;A9B3-A9C0M;A9CFL;A9D0-A9D9D;A9E0-A9E4L;A9E5M;A9E6-A9EFL"
";A9F0-A9F9D;A9FA-A9FEL;AA00-AA28L;AA29-AA36M;AA40-AA42L;AA43M;AA44-A"
"A4BL;AA4C-AA4DM;AA50-AA59D;AA60-AA76L;AA7AL;AA7B-AA7DM;AA7E-AAAFL;AA"
"B0M;AAB1L;AAB2-AAB4M;AAB5-AAB6L;AAB7-AAB8M;AAB9-AABDL;AABE-AABFM;AAC"
"0L;AAC1M;AAC2L;AADB-AADDL;AAE0-AAEAL;AAEB-AAEFM;AAF2-AAF4L;AAF5-AAF6"
"M;AB01-AB06L;AB09-AB0EL;AB11-AB16L;AB20-AB26L;AB28-AB2EL;AB30-AB5AL;"
"AB5C-AB69L;AB70-ABE2L;ABE3-ABEAM;ABEC-ABEDM;ABF0-ABF9D;AC00-D7A3L;D7"
"B0-D7C6L;D7CB-D7FBL;F900-FA6DL;FA70-FAD9L;FB00-FB06L;FB13-FB17L;FB1D"
"L;FB1EM;FB1F-FB28L;FB29S;FB2A-FB36L;FB38-FB3CL;FB3EL;FB40-FB41L;FB43"
"-FB44L;FB46-FBB1L;FBD3-FD3DL;FD50-FD8FL;FD92-FDC7L;FDF0-FDFBL;FDFCC;"
"FE00-FE0FM;FE20-FE2FM;FE31-FE32P;FE58P;FE62S;FE63P;FE64-FE66S;FE69C;"
"FE70-FE74L;FE76-FEFCL;FEFFF;FF04C;FF0BS;FF0DP;FF10-FF19D;FF1C-FF1ES;"
"FF21-FF3AL;FF41-FF5AL;FF5CS;FF5ES;FF66-FFBEL;FFC2-FFC7L;FFCA-FFCFL;F"
"FD2-FFD7L;FFDA-FFDCL;FFE0-FFE1C;FFE2S;FFE5-FFE6C;FFE9-FFECS;FFF9-FFF"
"BF;10000-1000BL;1000D-10026L;10028-1003AL;1003C-1003DL;1003F-1004DL;"
"10050-1005DL;10080-100FAL;10107-10133N;10140-10178N;1018A-1018BN;101"
"FDM;10280-1029CL;102A0-102D0L;102E0M;102E1-102FBN;10300-1031FL;10320"
"-10323N;1032D-10340L;10341N;10342-10349L;1034AN;10350-10375L;10376-1"
"037AM;10380-1039DL;103A0-103C3L;103C8-103CFL;103D1-103D5N;10400-1049"
"DL;104A0-104A9D;104B0-104D3L;104D8-104FBL;10500-10527L;10530-10563L;"
"10570-1057AL;1057C-1058AL;1058C-10592L;10594-10595L;10597-105A1L;105"
"A3-105B1L;105B3-105B9L;105BB-105BCL;10600-10736L;10740-10755L;10760-"
"10767L;10780-10785L;10787-107B0L;107B2-107BAL;10800-10805L;10808L;10"
"80A-10835L;10837-10838L;1083CL;1083F-10855L;10858-1085FN;10860-10876"
"L;10879-1087FN;10880-1089EL;108A7-108AFN;108E0-108F2L;108F4-108F5L;1"
"08FB-108FFN;10900-10915L;10916-1091BN;10920-10939L;10980-109B7L;109B"
"C-109BDN;109BE-109BFL;109C0-109CFN;109D2-109FFN;10A00L;10A01-10A03M;"
"10A05-10A06M;10A0C-10A0FM;10A10-10A13L;10A15-10A17L;10A19-10A35L;10A"
"38-10A3AM;10A3FM;10A40-10A48N;10A60-10A7CL;10A7D-10A7EN;10A80-10A9CL"
";10A9D-10A9FN;10AC0-10AC7L;10AC9-10AE4L;10AE5-10AE6M;10AEB-10AEFN;10"
"B00-10B35L;10B40-10B55L;10B58-10B5FN;10B60-10B72L;10B78-10B7FN;10B80"
"-10B91L;10BA9-10BAFN;10C00-10C48L;10C80-10CB2L;10CC0-10CF2L;10CFA-10"
"CFFN;10D00-10D23L;10D24-10D27M;10D30-10D39D;10E60-10E7EN;10E80-10EA9"
"L;10EAB-10EACM;10EADP;10EB0-10EB1L;10EFD-10EFFM;10F00-10F1CL;10F1D-1"
"0F26N;10F27L;10F30-10F45L;10F46-10F50M;10F51-10F54N;10F70-10F81L;10F"
"82-10F85M;10FB0-10FC4L;10FC5-10FCBN;10FE0-10FF6L;11000-11002M;11003-"
"11037L;11038-11046M;11052-11065N;11066-1106FD;11070M;11071-11072L;11"
"073-11074M;11075L;1107F-11082M;11083-110AFL;110B0-110BAM;110BDF;110C"
"2M;110CDF;110D0-110E8L;110F0-110F9D;11100-11102M;11103-11126L;11127-"
"11134M;11136-1113FD;11144L;11145-11146M;11147L;11150-11172L;11173M;1"
"1176L;11180-11182M;11183-111B2L;111B3-111C0M;111C1-111C4L;111C9-111C"
"CM;111CE-111CFM;111D0-111D9D;111DAL;111DCL;111E1-111F4N;11200-11211L"
";11213-1122BL;1122C-11237M;1123EM;1123F-11240L;11241M;11280-11286L;1"
"1288L;1128A-1128DL;1128F-1129DL;1129F-112A8L;112B0-112DEL;112DF-112E"
"AM;112F0-112F9D;11300-11303M;11305-1130CL;1130F-11310L;11313-11328L;"
"1132A-11330L;11332-11333L;11335-11339L;1133B-1133CM;1133DL;1133E-113"
"44M;11347-11348M;1134B-1134DM;11350L;11357M;1135D-11361L;11362-11363"
"M;11366-1136CM;11370-11374M;11400-11434L;11435-11446M;11447-1144AL;1"
"1450-11459D;1145EM;1145F-11461L;11480-114AFL;114B0-114C3M;114C4-114C"
"5L;114C7L;114D0-114D9D;11580-115AEL;115AF-115B5M;115B8-115C0M;115D8-"
"115DBL;115DC-115DDM;11600-1162FL;11630-11640M;11644L;11650-11659D;11"
"680-116AAL;116AB-116B7M;116B8L;116C0-116C9D;11700-1171AL;1171D-1172B"
"M;11730-11739D;1173A-1173BN;11740-11746L;11800-1182BL;1182C-1183AM;1"
"18A0-118DFL;118E0-118E9D;118EA-118F2N;118FF-11906L;11909L;1190C-1191"
"3L;11915-11916L;11918-1192FL;11930-11935M;11937-11938M;1193B-1193EM;"
"1193FL;11940M;11941L;11942-11943M;11950-11959D;119A0-119A7L;119AA-11"
"9D0L;119D1-119D7M;119DA-119E0M;119E1L;119E3L;119E4M;11A00L;11A01-11A"
"0AM;11A0B-11A32L;11A33-11A39M;11A3AL;11A3B-11A3EM;11A47M;11A50L;11A5"
"1-11A5BM;11A5C-11A89L;11A8A-11A99M;11A9DL;11AB0-11AF8L;11C00-11C08L;"
"11C0A-11C2EL;11C2F-11C36M;11C38-11C3FM;11C40L;11C50-11C59D;11C5A-11C"
"6CN;11C72-11C8FL;11C92-11CA7M;11CA9-11CB6M;11D00-11D06L;11D08-11D09L"
";11D0B-11D30L;11D31-11D36M;11D3AM;11D3C-11D3DM;11D3F-11D45M;11D46L;1"
"1D47M;11D50-11D59D;11D60-11D65L;11D67-11D68L;11D6A-11D89L;11D8A-11D8"
"EM;11D90-11D91M;11D93-11D97M;11D98L;11DA0-11DA9D;11EE0-11EF2L;11EF3-"
"11EF6M;11F00-11F01M;11F02L;11F03M;11F04-11F10L;11F12-11F33L;11F34-11"
"F3AM;11F3E-11F42M;11F50-11F59D;11FB0L;11FC0-11FD4N;11FDD-11FE0C;1200"
"0-12399L;12400-1246EN;12480-12543L;12F90-12FF0L;13000-1342FL;13430-1"
"343FF;13440M;13441-13446L;13447-13455M;14400-14646L;16800-16A38L;16A"
"40-16A5EL;16A60-16A69D;16A70-16ABEL;16AC0-16AC9D;16AD0-16AEDL;16AF0-"
"16AF4M;16B00-16B2FL;16B30-16B36M;16B40-16B43L;16B50-16B59D;16B5B-16B"
"61N;16B63-16B77L;16B7D-16B8FL;16E40-16E7FL;16E80-16E96N;16F00-16F4AL"
";16F4FM;16F50L;16F51-16F87M;16F8F-16F92M;16F93-16F9FL;16FE0-16FE1L;1"
"6FE3L;16FE4M;16FF0-16FF1M;17000-187F7L;18800-18CD5L;18D00-18D08L;1AF"
"F0-1AFF3L;1AFF5-1AFFBL;1AFFD-1AFFEL;1B000-1B122L;1B132L;1B150-1B152L"
";1B155L;1B164-1B167L;1B170-1B2FBL;1BC00-1BC6AL;1BC70-1BC7CL;1BC80-1B"
"C88L;1BC90-1BC99L;1BC9D-1BC9EM;1BCA0-1BCA3F;1CF00-1CF2DM;1CF30-1CF46"
"M;1D165-1D169M;1D16D-1D172M;1D173-1D17AF;1D17B-1D182M;1D185-1D18BM;1"
"D1AA-1D1ADM;1D242-1D244M;1D2C0-1D2D3N;1D2E0-1D2F3N;1D360-1D378N;1D40"
"0-1D454L;1D456-1D49CL;1D49E-1D49FL;1D4A2L;1D4A5-1D4A6L;1D4A9-1D4ACL;"
"1D4AE-1D4B9L;1D4BBL;1D4BD-1D4C3L;1D4C5-1D505L;1D507-1D50AL;1D50D-1D5"
"14L;1D516-1D51CL;1D51E-1D539L;1D53B-1D53EL;1D540-1D544L;1D546L;1D54A"
"-1D550L;1D552-1D6A5L;1D6A8-1D6C0L;1D6C1S;1D6C2-1D6DAL;1D6DBS;1D6DC-1"
"D6FAL;1D6FBS;1D6FC-1D714L;1D715S;1D716-1D734L;1D735S;1D736-1D74EL;1D"
"74FS;1D750-1D76EL;1D76FS;1D770-1D788L;1D789S;1D78A-1D7A8L;1D7A9S;1D7"
"AA-1D7C2L;1D7C3S;1D7C4-1D7CBL;1D7CE-1D7FFD;1DA00-1DA36M;1DA3B-1DA6CM"
";1DA75M;1DA84M;1DA9B-1DA9FM;1DAA1-1DAAFM;1DF00-1DF1EL;1DF25-1DF2AL;1"
"E000-1E006M;1E008-1E018M;1E01B-1E021M;1E023-1E024M;1E026-1E02AM;1E03"
"0-1E06DL;1E08FM;1E100-1E12CL;1E130-1E136M;1E137-1E13DL;1E140-1E149D;"
"1E14EL;1E290-1E2ADL;1E2AEM;1E2C0-1E2EBL;1E2EC-1E2EFM;1E2F0-1E2F9D;1E"
"2FFC;1E4D0-1E4EBL;1E4EC-1E4EFM;1E4F0-1E4F9D;1E7E0-1E7E6L;1E7E8-1E7EB"
"L;1E7ED-1E7EEL;1E7F0-1E7FEL;1E800-1E8C4L;1E8C7-1E8CFN;1E8D0-1E8D6M;1"
"E900-1E943L;1E944-1E94AM;1E94BL;1E950-1E959D;1EC71-1ECABN;1ECAD-1ECA"
"FN;1ECB0C;1ECB1-1ECB4N;1ED01-1ED2DN;1ED2F-1ED3DN;1EE00-1EE03L;1EE05-"
"1EE1FL;1EE21-1EE22L;1EE24L;1EE27L;1EE29-1EE32L;1EE34-1EE37L;1EE39L;1"
"EE3BL;1EE42L;1EE47L;1EE49L;1EE4BL;1EE4D-1EE4FL;1EE51-1EE52L;1EE54L;1"
"EE57L;1EE59L;1EE5BL;1EE5DL;1EE5FL;1EE61-1EE62L;1EE64L;1EE67-1EE6AL;1"
"EE6C-1EE72L;1EE74-1EE77L;1EE79-1EE7CL;1EE7EL;1EE80-1EE89L;1EE8B-1EE9"
"BL;1EEA1-1EEA3L;1EEA5-1EEA9L;1EEAB-1EEBBL;1EEF0-1EEF1S;1F100-1F10CN;"
"1FBF0-1FBF9D;20000-2A6DFL;2A700-2B739L;2B740-2B81DL;2B820-2CEA1L;2CE"
"B0-2EBE0L;2EBF0-2EE5DL;2F800-2FA1DL;30000-3134AL;31350-323AFL;E0001F"
";E0020-E007FF;E0100-E01EFM"
)
# END UNICODE TABLE
_UT = []
def _utable():
if not _UT:
if hashlib.sha256(_UTABLE.encode("ascii")).hexdigest() != \
UNICODE_TABLE_SHA256:
raise RuntimeError("unicode table")
starts, ends, cls = [], [], []
for item in _UTABLE.split(";"):
rng, c = item[:-1], item[-1]
a, _, b = rng.partition("-")
starts.append(int(a, 16))
ends.append(int(b or a, 16))
cls.append(c)
_UT.extend([starts, ends, cls])
return _UT
def ucls(ch):
"""The table class of one character, or "" for none."""
starts, ends, cls = _utable()
cp = ord(ch)
k = bisect.bisect_right(starts, cp) - 1
return cls[k] if k >= 0 and cp <= ends[k] else ""
# ------------------------------------------------------------ quote check
_SEP = (".", ",", "'", "\N{RIGHT SINGLE QUOTATION MARK}", "_")
_JOIN3X = ("\N{MINUS SIGN}", "/", ":", "~", "\N{FULLWIDTH TILDE}",
"\N{FRACTION SLASH}", "\N{DIVISION SLASH}", "\N{RATIO}",
"\N{DIVISION SIGN}", "\N{BIG SOLIDUS}", "\N{SWUNG DASH}")
_VULGAR = tuple(chr(c) for c in list(range(0xBC, 0xBF))
+ list(range(0x2150, 0x2160)) + [0x2189])
_FRAC_SL = ("/", "\N{FRACTION SLASH}", "\N{DIVISION SLASH}")
_MULT = ("*", "\N{MULTIPLICATION SIGN}", "\N{MIDDLE DOT}")
_SUPSIGN = ("\N{SUPERSCRIPT PLUS SIGN}", "\N{SUPERSCRIPT MINUS}")
_AFFIXES = ("%", "\N{PER MILLE SIGN}", "\N{PER TEN THOUSAND SIGN}",
"\N{DEGREE SIGN}", "\N{PRIME}", "\N{DOUBLE PRIME}",
"\N{FULLWIDTH PERCENT SIGN}", "\N{SMALL PERCENT SIGN}",
"\N{ARABIC PERCENT SIGN}", "\N{ARABIC-INDIC PER MILLE SIGN}",
"\N{ARABIC-INDIC PER TEN THOUSAND SIGN}")
_AFFIX_WORDS = ("percent", "pct")
_AFFIX_BOTH = ("percent", "pct", "per cent")
_AFFIX_AFTER = ("permille", "bp", "bps", "pp", "ppm", "ppb", "ppt", "ppq")
_TWO_WORDS = (("per", "cent"), ("per", "mille"), ("basis", "point"),
("basis", "points"), ("percentage", "point"),
("percentage", "points"))
_AFFIX_DECL = ("percent", "per cent", "pct", "per mille", "permille", "bp",
"bps", "basis point", "basis points", "pp",
"percentage point", "percentage points", "ppm", "ppb", "ppt",
"ppq")
_SCALE_WORDS = ("hundred", "thousand", "million", "billion", "trillion",
"quadrillion", "lakh", "crore", "dozen", "tenth",
"hundredth", "thousandth", "millionth", "billionth",
"trillionth")
_SCALE_CI = ("mn", "bn", "mln", "bln", "tn", "trn", "mio", "mrd")
_SCALE_ABBR = ("k", "K", "M", "m", "B", "MM")
_SCALE_DECL = _SCALE_WORDS + _SCALE_CI + _SCALE_ABBR
_ACC_CH = "0123456789.,'_\N{RIGHT SINGLE QUOTATION MARK}"
def _alow(w):
"""ASCII lower case, or None for a word that is not ASCII."""
for ch in w:
if ord(ch) > 127:
return None
return w.lower()
def _prep(s):
"""Cf characters removed, then canon_text."""
if type(s) is not str:
raise ValueError("text")
return canon_text("".join(ch for ch in s if ucls(ch) != "F"))
def _acc(s):
"""TK7's accounting reading: "(" number ")" becomes "-" number."""
out = []
n = len(s)
k = 0
while k < n:
if s[k] == "(":
e = k + 1
while e < n and (s[e] in _ACC_CH or ucls(s[e]) == "Z"):
e += 1
b = s[k + 1:e]
if (e < n and s[e] == ")" and b and b[-1] in "0123456789"
and (b[0] in "0123456789" or (
len(b) > 1 and b[0] in ".," and b[1] in "0123456789"))):
out.append("-" + b)
k = e + 1
continue
out.append(s[k])
k += 1
return "".join(out)
def _classes(h):
cl = [ucls(c) for c in h]
return {"alnum": [c in ("L", "M", "D", "N") for c in cl],
"num": [c in ("D", "N") for c in cl],
"nd": [c == "D" for c in cl],
"sign": [c in ("P", "S") for c in cl],
"j3": [cl[k] == "P" or h[k] in _JOIN3X
for k in range(len(h))],
"zs": [c == "Z" for c in cl],
"let": [c in ("L", "M") for c in cl],
"sym": [h[k] in _AFFIXES or cl[k] == "C" for k in range(len(h))]}
def _joins(h, K):
"""J[g] for each gap g (between h[g-1] and h[g]): True where both
characters lie in one word or number token (TK0-TK8)."""
n = len(h)
alnum, num, nd = K["alnum"], K["num"], K["nd"]
sign, zs, let, sym = K["sign"], K["zs"], K["let"], K["sym"]
j3 = K["j3"]
J = [False] * (n + 1)
def D(k):
return 0 <= k < n and num[k]
def lead(k):
return D(k) or (0 <= k < n and h[k] in ".," and D(k + 1))
def fz(k):
while k < n and zs[k]:
k += 1
return k
def bz(k):
while k >= 0 and zs[k]:
k -= 1
return k
def run(k):
while k < n and let[k]:
k += 1
return k
def link(a, b):
for g in range(a, b + 1):
J[g] = True
for k in range(n):
c = h[k]
if k >= 1 and alnum[k - 1] and alnum[k]:
J[k] = True
if c in _SEP and D(k - 1) and D(k + 1):
link(k, k + 1)
if c in ".," and D(k + 1):
J[k + 1] = True
if zs[k] and D(k - 1):
t = fz(k)
if D(t) and D(t + 1) and D(t + 2) and not D(t + 3):
link(k, t)
if j3[k]:
a, b = bz(k - 1), fz(k + 1)
if D(a) and D(b):
link(a + 1, b)
if c in "eE" and D(k - 1) and k + 1 < n and sign[k + 1] and D(k + 2):
J[k + 1] = True
if c == "^" and D(k - 1):
J[k] = True
if D(k + 1) or (k + 1 < n and sign[k + 1] and D(k + 2)):
J[k + 1] = True
if c in _SUPSIGN and D(k - 1):
J[k] = True
for d in range(n):
if not D(d) or D(d + 1):
continue
k = fz(d + 1)
if d + 1 < k < n:
if h[k] in _VULGAR:
link(d + 1, k)
elif D(k):
e = k
while D(e):
e += 1
if e < n and h[e] in _FRAC_SL and D(e + 1):
link(d + 1, k)
if k < n:
k2 = -1
if h[k] in _MULT:
k2 = k + 1
elif h[k] in "xX" and k > d + 1 and k + 1 < n and zs[k + 1]:
k2 = k + 1
if k2 >= 0:
k3 = fz(k2)
if h[k3:k3 + 2] == "10" and not (k3 + 2 < n and nd[k3 + 2]):
link(d + 1, k3 + 1)
t = k
if t >= n:
continue
if sym[t]:
link(d + 1, t)
if h[t:t + 2] == "**":
q = fz(t + 2)
if q < n and sign[q]:
q += 1
if D(q):
link(d + 1, q)
if not let[t]:
continue
e = run(t)
w = h[t:e]
lw = _alow(w)
whole = e == n or not alnum[e]
if w in ("e", "E"):
q = fz(e)
if q < n and sign[q]:
q += 1
if D(q):
link(d + 1, q)
if not whole or lw is None:
continue
if (lw in _AFFIX_WORDS or lw in _AFFIX_AFTER or lw in _SCALE_WORDS
or lw in _SCALE_CI or w in _SCALE_ABBR
or (lw[-1:] == "s" and lw[:-1] in _SCALE_WORDS)):
link(d + 1, e - 1)
elif lw in ("per", "basis", "percentage"):
t2 = fz(e)
if t2 > e and t2 < n and let[t2]:
e2 = run(t2)
if ((lw, _alow(h[t2:e2])) in _TWO_WORDS
and (e2 == n or not alnum[e2])):
link(d + 1, e2 - 1)
elif lw == "times":
t2 = fz(e)
if t2 > e and h[t2:t2 + 3] == "10^":
link(d + 1, t2 + 2)
for s in range(n):
if not let[s] or (s > 0 and alnum[s - 1]):
continue
e = run(s)
lw = _alow(h[s:e])
ok = lw in _AFFIX_WORDS
if lw == "cent":
b = bz(s - 1)
if b < s - 1 and b >= 0 and let[b]:
bs = b
while bs > 0 and let[bs - 1]:
bs -= 1
ok = _alow(h[bs:b + 1]) == "per"
if ok:
t = fz(e)
if lead(t):
link(e, t)
for k in range(n - 1, -1, -1):
if sign[k]:
if lead(k + 1) or (k + 1 < n and (sign[k + 1] or sym[k + 1])
and J[k + 2]):
J[k + 1] = True
else:
t = fz(k + 1)
if t > k + 1 and (lead(t) or (t < n and (sign[t] or sym[t])
and J[t + 1])):
if not D(bz(k - 1)):
link(k + 1, t)
elif sym[k]:
t = fz(k + 1)
if lead(t) or (t < n and sign[t] and J[t + 1]):
link(k + 1, t)
return J
def _check_affix(affix):
if affix is None or (type(affix) is str and affix in _AFFIX_DECL):
return
if (type(affix) is not str or not 1 <= len(affix) <= 4
or not (affix[0] in _AFFIXES or ucls(affix[0]) == "C")):
raise ValueError("affix")
for ch in affix[1:]:
if ucls(ch) != "L":
raise ValueError("affix")
def _check_scale(scale):
if scale is not None and (type(scale) is not str
or scale not in _SCALE_DECL):
raise ValueError("scale")
def _form_end(h, t, form, K):
"""End of the declared affix or scale standing at t, or -1."""
n = len(h)
let, zs = K["let"], K["zs"]
if form in _AFFIX_DECL or form in _SCALE_DECL:
if t >= n or not let[t]:
return -1
e = t
while e < n and let[e]:
e += 1
w = h[t:e]
if form in _SCALE_ABBR:
return e if w == form else -1
lw = _alow(w)
if " " in form:
first, second = form.split(" ")
t2 = e
while t2 < n and zs[t2]:
t2 += 1
if lw != first or t2 == e or t2 >= n or not let[t2]:
return -1
e2 = t2
while e2 < n and let[e2]:
e2 += 1
lw2 = _alow(h[t2:e2])
if lw2 == second or (second in ("point", "points")
and lw2 in ("point", "points")):
return e2
return -1
if form in _SCALE_WORDS:
return e if lw in (form, form + "s") else -1
return e if lw == form else -1
return t + len(form) if h[t:t + len(form)] == form else -1
def _suffix_ok(h, j, affix, scale, J, K):
n = len(h)
zs, alnum = K["zs"], K["alnum"]
forms = [f for f in (scale, affix) if f is not None]
pos = j
took = False
for _ in range(2):
t = pos
while t < n and zs[t]:
t += 1
hit = -1
for f in forms:
e = _form_end(h, t, f, K)
if e >= 0:
hit = e
forms.remove(f)
break
if hit < 0:
break
pos = hit
took = True
return took and (pos == n or (not alnum[pos] and not J[pos]))
def _prefix_ok(h, i, affix, J, K):
zs, let, alnum = K["zs"], K["let"], K["alnum"]
t = i
while t > 0 and zs[t - 1]:
t -= 1
if affix in _AFFIX_DECL:
if affix not in _AFFIX_BOTH:
return False
s = t
while s > 0 and let[s - 1]:
s -= 1
if s == t:
return False
lw = _alow(h[s:t])
if affix == "per cent":
b = s
while b > 0 and zs[b - 1]:
b -= 1
s2 = b
while s2 > 0 and let[s2 - 1]:
s2 -= 1
if lw != "cent" or b == s or s2 == b or _alow(h[s2:b]) != "per":
return False
s = s2
elif lw != affix:
return False
else:
s = t - len(affix)
if s < 0 or h[s:t] != affix:
return False
return s == 0 or (not alnum[s - 1] and not J[s])
def _check_flags(affix, scale, accounting):
_check_affix(affix)
_check_scale(scale)
if type(accounting) is not bool:
raise ValueError("accounting")
def falcon_quote_match(value, quote_text, affix=None, scale=None,
accounting=False):
"""PASS if the value occurs in the quote text as a whole token under the
TOKEN RULE (TK0-TK11 in the module docstring), else FAIL."""
v = _prep(value)
h = _prep(quote_text)
_check_flags(affix, scale, accounting)
if accounting:
v = _acc(v)
h = _acc(h)
if v == "":
return "FAIL"
n = len(h)
K = _classes(h)
J = _joins(h, K)
alnum = K["alnum"]
vf, vl = ucls(v[0]), ucls(v[-1])
pre_ok = vf in ("D", "N", "P", "S") or v[0] in ".,"
post_ok = vl in ("D", "N")
i = h.find(v)
while i >= 0:
j = i + len(v)
left = i == 0 or (not alnum[i - 1] and not J[i])
right = j == n or (not alnum[j] and not J[j])
if not right and post_ok and (affix is not None or scale is not None):
right = _suffix_ok(h, j, affix, scale, J, K)
if not left and pre_ok and affix is not None:
left = _prefix_ok(h, i, affix, J, K)
if left and right:
return "PASS"
i = h.find(v, i + 1)
return "FAIL"
def quote_check(page_text, quote, value, affix=None, scale=None,
accounting=False):
"""TK11: UNKNOWN without page text; PASS when the quote is a whole token
of the page text and the value a whole token of the quote; else FAIL."""
_prep(quote)
_prep(value)
_check_flags(affix, scale, accounting)
if page_text is None:
return "UNKNOWN"
_prep(page_text)
if (falcon_quote_match(quote, page_text, affix, scale, accounting)
== "PASS" and falcon_quote_match(value, quote, affix, scale,
accounting) == "PASS"):
return "PASS"
return "FAIL"
def render_quote(q):
return "quote|" + q
# ----------------------------------------------------------- source match
def _url_parts(c):
"""Parts of a URL already in canon_url's form."""
scheme, auth, path, query, frag = _APPB.fullmatch(c).groups()
ui = host = None
port = ""
if auth is not None:
hp = auth
if "@" in auth:
ui, hp = auth.split("@", 1)
if hp[:1] == "[":
k = hp.find("]")
host, port = hp[:k + 1], hp[k + 1:]
else:
k = hp.find(":")
host, port = (hp, "") if k < 0 else (hp[:k], hp[k:])
return scheme, ui, host, port, path, query, frag
def _host_ok(host):
return (host is not None and host != "" and host[:1] != "["
and re.fullmatch(r"[0-9.]+", host) is None and "%" not in host)
_PCT_ANY = re.compile(r"%([0-9A-Fa-f]{2})")
_ENC_SEP = re.compile(r"%(?:2[Ff]|5[Cc])")
_HOSTNAME = re.compile(r"[a-z0-9](?:[a-z0-9.-]*[a-z0-9])?")
_TEMPLATE = re.compile(r"(?:/[A-Za-z0-9._~!$&'()*+,=:@-]+)+/")
def _path_bad(path):
"""SM2: after percent-decoding repeated until stable (at most seven
levels), a dot-segment or a ";"; at any level an encoded "/", a
backslash or bytes that are not valid UTF-8 (N4); or decoding that
does not settle."""
cur = path
for _ in range(8):
if (_ENC_SEP.search(cur) is not None or "\\" in cur
or not _utf8_ok(cur)):
return True
nxt = _PCT_ANY.sub(lambda m: chr(int(m.group(1), 16)), cur)
if nxt == cur:
break
cur = nxt
else:
return True
return ";" in cur or any(s in (".", "..") for s in cur.split("/"))
def _utf8_ok(cur):
"""N4: the bytes of one decoding stage form valid UTF-8 (RFC 3629: no
overlong form, surrogate or truncated sequence)."""
try:
cur.encode("latin-1").decode("utf-8")
except UnicodeError:
return False
return True
def _tmpl_ok(segs, tmpl):
"""segs reach the template; its last "*", the branch or commit slot,
is not "refs" in any ASCII case (N2)."""
if len(segs) < len(tmpl):
return False
for t, s in zip(tmpl, segs):
if not ((t == "*" and s != "") or t == s):
return False
stars = [i for i, t in enumerate(tmpl) if t == "*"]
return not stars or _alow(segs[stars[-1]]) != "refs"
def _check_sources(source_list):
"""The entries of the list in order (None for a CODE HOST mark)."""
if type(source_list) is not list:
raise ValueError("sources")
code = {}
for e in source_list:
if type(e) is list and len(e) >= 2 and e[0] == "CODE HOST":
h = e[1]
if (type(h) is not str or _HOSTNAME.fullmatch(h) is None
or re.fullmatch(r"[0-9.]+", h) is not None or h in code):
raise ValueError("sources")
tl = []
for tp in e[2:]:
if type(tp) is not str or _TEMPLATE.fullmatch(tp) is None:
raise ValueError("sources")
tl.append(tp.strip("/").split("/"))
code[h] = tl
out = []
for e in source_list:
if type(e) is not list or len(e) < 2:
raise ValueError("sources")
if e[0] == "CODE HOST":
out.append(None)
continue
if len(e) != 2 or e[0] not in ("EXACT", "PREFIX", "SERVED"):
raise ValueError("sources")
try:
c = canon_url(e[1])
except ValueError:
raise ValueError("sources")
scheme, ui, host, port, path, query, frag = _url_parts(c)
if (scheme != "https" or host is None or ui is not None
or not _host_ok(host) or frag is not None
or re.fullmatch(r"(?::[0-9]+)?", port) is None
or (e[0] != "SERVED"
and _path_bad(_APPB.fullmatch(e[1]).group(3)))
or (e[0] == "PREFIX" and query is not None)):
raise ValueError("sources")
path = path or "/"
if e[0] == "PREFIX":
if path == "/":
raise ValueError("publisher")
if host in code:
segs = path.strip("/").split("/")
if not any(_tmpl_ok(segs, t) for t in code[host]):
raise ValueError("publisher")
out.append({"kind": e[0], "host": host, "port": port, "path": path,
"query": query,
"url": "https://" + host + port + path
+ ("" if query is None else "?" + query)})
return out
def _match(cited_url, entries, hop=False):
"""SM3-SM8 and SM10; hop skips SERVED entries (SM9)."""
if type(cited_url) is not str:
return {"status": "UNKNOWN", "reason": "url"}
try:
c = canon_url(cited_url)
except ValueError:
return {"status": "UNKNOWN", "reason": "url"}
scheme, ui, host, port, path, query, frag = _url_parts(c)
if host is None:
return {"status": "UNKNOWN", "reason": "url"}
if scheme != "https":
return {"status": "UNKNOWN", "reason": "scheme"}
if ui is not None:
return {"status": "UNKNOWN", "reason": "userinfo"}
if not _host_ok(host):
return {"status": "UNKNOWN", "reason": "host"}
bad = _path_bad(_APPB.fullmatch(cited_url).group(3))
path = path or "/"
best = None
if not bad:
for i, e in enumerate(entries):
if e is None or e["host"] != host or e["port"] != port:
continue
if e["kind"] == "EXACT":
if e["path"] == path and (e["query"] is None
or e["query"] == query):
return {"status": "ENTRY", "index": i, "kind": "EXACT",
"declared": e["url"], "fetch": e["url"]}
if not hop:
for i, e in enumerate(entries):
if e is None or e["kind"] != "SERVED" or e["host"] != host \
or e["port"] != port:
continue
if e["path"] == path and e["query"] == query:
return {"status": "ENTRY", "index": i, "kind": "SERVED",
"declared": e["url"], "fetch": "STORE"}
if bad:
return {"status": "UNKNOWN", "reason": "path"}
for i, e in enumerate(entries):
if e is None or e["kind"] != "PREFIX" or e["host"] != host \
or e["port"] != port:
continue
p = e["path"]
under = path[:len(p)] == p and (
p[-1:] == "/" or path[len(p):len(p) + 1] == "/")
if path == p or under:
if best is None or len(p) > len(entries[best]["path"]):
best = i
if best is None:
return {"status": "UNKNOWN", "reason": "none"}
e = entries[best]
return {"status": "ENTRY", "index": best, "kind": "PREFIX",
"declared": e["url"], "fetch": "https://" + host + port + path}
def source_match(cited_url, source_list):
"""SM1-SM8 in the module docstring."""
return _match(cited_url, _check_sources(source_list))
def source_hop(hop_url, matched, source_list):
"""SM9: a redirect hop matched again at entry level; stays tells whether
it lands in the entry the citation matched."""
entries = _check_sources(source_list)
if (not _is_int(matched, 0, len(entries) - 1)
or entries[matched] is None
or entries[matched]["kind"] == "SERVED"):
raise ValueError("matched")
d = _match(hop_url, entries, True)
d["stays"] = d["status"] == "ENTRY" and d["index"] == matched
return d
def render_source(d):
if d["status"] == "UNKNOWN":
return "source|UNKNOWN|" + d["reason"]
return "\n".join(["source|ENTRY|%d|%s|%s" % (d["index"], d["kind"],
_esc(d["declared"])),
"fetch|" + _esc(d["fetch"])])
def render_hop(d):
tail = ("hop|END" if d["status"] == "UNKNOWN"
else "hop|SAME" if d["stays"] else "hop|OTHER")
return render_source(d) + "\n" + tail
# -------------------------------------------------------------- kind stop
def _breaker(s):
"""FALCON BREAKER as ten-thousandths, from "0", "0.3", ... "1"."""
if type(s) is not str or re.fullmatch(
r"0(?:\.[0-9]{1,4})?|1(?:\.0{1,4})?", s) is None:
raise ValueError("limits")
if s[0] == "1":
return 10000
return int((s[2:] + "0000")[:4])
def falcon_stop(counts, limits):
"""SP1-SP6 in the module docstring."""
if not _keys_are(limits, ("breaker", "window")):
raise ValueError("limits")
b4 = _breaker(limits["breaker"])
if not _is_int(limits["window"], 1, 1000):
raise ValueError("limits")
wmax = limits["window"]
if type(counts) is not list:
raise ValueError("counts")
stopped = False
epoch = 0
flying = {}
closed = set()
win = []
alone = None
for e in counts:
if type(e) is not list or not e:
raise ValueError("counts")
if e[0] == "ISSUED":
if (len(e) != 2 or not _is_name(e[1]) or e[1] in flying
or e[1] in closed or stopped):
raise ValueError("counts")
if alone is not None:
if alone != "" or flying:
raise ValueError("counts")
alone = e[1]
flying[e[1]] = epoch
elif e[0] == "FINISHED":
if len(e) != 5 or not _is_name(e[1]) or e[1] not in flying:
raise ValueError("counts")
for x in e[2:]:
if not _is_int(x, 0, COUNT_MAX):
raise ValueError("counts")
if e[3] + e[4] > e[2]:
raise ValueError("counts")
ep = flying.pop(e[1])
closed.add(e[1])
if ep != epoch:
continue
win.append((e[2], e[3] + e[4]))
if alone == e[1]:
alone = None
if not stopped:
total = sum(x[0] for x in win[-wmax:])
bad = sum(x[1] for x in win[-wmax:])
if total > 0 and bad * 10000 > b4 * total:
stopped = True
elif e[0] == "RESUMED":
if len(e) != 1:
raise ValueError("counts")
if stopped:
stopped = False
epoch += 1
win = []
alone = ""
else:
raise ValueError("counts")
if stopped:
status = "STOPPED"
elif alone is not None and (alone != "" or flying):
status = "STOPPED"
else:
status = "ISSUING"
w = win[-wmax:]
return {"status": status, "bad": sum(x[1] for x in w),
"total": sum(x[0] for x in w), "falcons": len(w),
"alone": alone, "in_flight": len(flying)}
def render_stop(d):
a = d["alone"]
return "\n".join([
"stop|" + d["status"],
"share|%d|%d|%d" % (d["bad"], d["total"], d["falcons"]),
"alone|" + ("-" if a is None else "waiting" if a == "" else _esc(a)),
"in_flight|%d" % d["in_flight"]])
# ------------------------------------------------ output-schema digest (0.2.0)
SCHEMA_TAG = "TPC-FALCON-SCHEMA"
SCHEMA_FORM = "1"
def _schema_norm(v, top):
"""FS2 over a checked schema: narrow dropped at the top; quote false
and confidential false dropped wherever they stand."""
if type(v) is dict:
out = {}
for k in v:
if top and k == "narrow":
continue
if k in ("quote", "confidential") and v[k] is False:
continue
out[k] = _schema_norm(v[k], False)
return out
if type(v) is list:
return [_schema_norm(x, False) for x in v]
return v
def _schema_str(s):
"""FS3's string: LF line ends, then NFC, quoted and escaped; FS1 0.2.0
(D213.1): a string not already its canonical text is refused."""
for ch in s:
if 0xD800 <= ord(ch) <= 0xDFFF:
raise ValueError("schema")
if "\r" in s or not unicodedata.is_normalized("NFC", s):
raise ValueError("schema")
s = unicodedata.normalize("NFC", s.replace("\r\n", "\n").replace("\r", "\n"))
out = ['"']
for ch in s:
if ch == '"' or ch == "\\":
out.append("\\" + ch)
elif ord(ch) < 0x20:
out.append("\\u%04x" % ord(ch))
else:
out.append(ch)
out.append('"')
return "".join(out)
def _schema_text(v):
if v is True:
return "true"
if v is False:
return "false"
if type(v) is str:
return _schema_str(v)
if type(v) is list:
return "[" + ",".join(_schema_text(x) for x in v) + "]"
if type(v) is dict:
return "{" + ",".join(_schema_str(k) + ":" + _schema_text(v[k])
for k in sorted(v)) + "}"
raise ValueError("schema") # unreachable: the check admits no other value
def falcon_schema_canon(schema):
try:
_check_schema(schema)
return ["OK", _schema_text(_schema_norm(schema, True))]
except ValueError:
return ["REFUSED", "schema"]
def falcon_schema_digest(schema):
r = falcon_schema_canon(schema)
if r[0] != "OK":
return r
return ["OK", _digest_fields([SCHEMA_TAG, SCHEMA_FORM, r[1]])]
def render_schema(r, tag):
return tag + "|" + r[1] if r[0] == "OK" else "ERROR:" + r[1]
# ------------------------------------------------------------ corpus case
FNS = ("canon", "draw", "reconcile", "stage", "qmatch", "qcheck", "source",
"hop", "stop", "schema_canon", "schema_digest")
def run_case(fn, args):
"""One corpus case: the render, or "ERROR:" and the message."""
if fn not in FNS:
raise ValueError("fn")
try:
if fn == "canon":
return render_canon(falcon_canon(args["value"], args["spec"]))
if fn == "draw":
return render_draw(falcon_draw(args["candidates"], args["n"],
args["state"], args["r"]))
if fn == "reconcile":
return render_reconcile(falcon_reconcile(
args["outputs"], args["quotes"], args["schema"]))
if fn == "stage":
return render_stage(sponsor_stage(args["events"], args["as_of"],
args["thresholds"]))
if fn == "qmatch":
return render_quote(falcon_quote_match(
args["value"], args["quote"], args.get("affix"),
args.get("scale"), args.get("accounting", False)))
if fn == "qcheck":
return render_quote(quote_check(
args["page"], args["quote"], args["value"],
args.get("affix"), args.get("scale"),
args.get("accounting", False)))
if fn == "source":
return render_source(source_match(args["cited"],
args["sources"]))
if fn == "hop":
return render_hop(source_hop(args["hop"], args["matched"],
args["sources"]))
if fn == "schema_canon":
return render_schema(falcon_schema_canon(args["schema"]), "canon")
if fn == "schema_digest":
return render_schema(falcon_schema_digest(args["schema"]), "digest")
return render_stop(falcon_stop(args["counts"], args["limits"]))
except ValueError as ex:
return "ERROR:" + str(ex)