The Principia Machina/TPC-PY/tpc digest.py: Difference between revisions

Latest revision as of 13:34, 30 September 2026

# -*- coding: utf-8 -*-
"""
TPC-PY --- digest primitive and the D(TFK) combination rule
===========================================================
Discharges the two OWED items in TPS 2.5.3 at the primitive + combination
level (C-02 the convention, C-03 the cross-implementation hash tests):

  OWED (C-02): the combination rule for D(TFK) over the three zone digests,
               fixed order, fixed algorithm, stated once, so independent
               verifiers agree.
  OWED (C-03): cross-implementation hash tests, since serialization drift
               is the known failure of hash chains.

STATUS: RATIFIED (decision D97.1, 2026-09-22). The convention is fixed and
digest VALUES are VERIFIED. Its text still lands in TPS 2.5 at revision 5
(owed spec edit). Standard-library only; Python 3.8+; Windows/Linux.

0.2.0 (proposed with TPC-PY 0.2.0; D203.1, taken 2026-09-28: the pause
tpm_post_tpcpy_pause 0.1.0, D203 Option 1 (a) and (b); tpm_tps_digest_spec
0.1.0): the TPS digest D(TPS) and the instruction set digest D(IS), over
the <pre> text of the declared TPS tree's pages. digest_fields,
combine_digests and d_tfk are unchanged.

0.3.0 (proposed with TPC-PY 0.3.0; D216.1, taken 2026-09-29: the pause
tpm_post_tpcpy_wave4_pause 0.2.0, D216 Option 1 (c); tps_pages_definition
0.1.0): tps_pages, the ordered list [title, h] of the kept pages, the very
list d_tps digests, which TPX publishes beside D(TPS) and D(IS) at each TPS
declaration and against which PRB names the pages that differ. d_tps is
built on it (the version checked, then the list digested), so no answer of
d_tps moves; pre_texts, d_is, digest_fields, combine_digests and d_tfk are
unchanged.

  tps_pages(pages) -> ["OK", [[title, h], ...]] or ["REFUSED", reason]
    pages as d_tps takes them. The list: the kept pages (the root and the
    section pages with at least one block; the contents page never), in
    d_tps's order, each with h, the SHA-256 of its blocks joined by one
    LF, as UTF-8, in lowercase hex; nothing normalized; [] when no page is
    kept. Reasons: d_tps's less "version", in its order ("pages",
    "title", "root", "pre"), the first that applies whatever the order of
    the pages. d_tps(version, pages) refuses "version" first and otherwise
    exactly when tps_pages(pages) refuses, with the same reason; when it
    answers OK, its digest is digest_fields(["TPC-TPS", "1", version, n,
    title_1, h_1, ..., title_n, h_n]) over this list.

  pre_texts(wikitext) -> ["OK", [block, ...]] or ["REFUSED", "pre"]
    A block opens at the exact text "<pre>" followed by LF and ends at the
    next exact "</pre>"; the block is the text between, as it stands.
    A PRE TAG is "<pre" or "</pre" (ASCII letters in any case) followed by
    ">", "/", ASCII white space (space, tab, LF, VT, FF, CR) or the end of
    the text; "<prefix>" is text. Refused: a pre tag of any other form
    (another case, an attribute, white space, "/>"), an opening not
    followed by LF, an opening inside a block, a closing outside one, an
    unclosed block, a wikitext that is not a string or has no UTF-8 form.
    Comments and nowiki are not parsed: a pre tag inside either is read
    as a tag (a named limit). No block: ["OK", []].
  d_tps(version, pages) -> ["OK", hex] or ["REFUSED", reason]
    pages: [title, wikitext] pairs, every page read of the TPS tree, in
    any order. Titles in MediaWiki's text form: the root TPS_ROOT; the
    contents page TPS_ROOT + "/Contents", left out by title; a section
    page TPS_ROOT + "/" + number + " " + name, the number two or more
    parts joined by ".", each part 0 or 1 to 4 digits without a leading
    zero, the name without "/", without a leading, trailing or doubled
    space. Kept: the root and the section pages that carry at least one
    block; ordered: the root, then the sections by number compared as
    integers part by part (5.9 before 5.10; 5.1 before 5.1.2 before 5.2).
    h = SHA-256 of the page's blocks joined by one LF, as UTF-8, in
    lowercase hex. D(TPS) = digest_fields(["TPC-TPS", "1", version, n,
    title_1, h_1, ..., title_n, h_n]), n an integer.
    Reasons, the first that applies whatever the order of the pages:
    "version" (not MAJOR.MINOR.PATCH, ASCII digits, no leading zero);
    "pages" (not a list of [string, string]); "title" (a title outside
    the forms above, holding "_" or a control character, or not UTF-8; a
    title or a section number twice); "root" (no page titled TPS_ROOT);
    "pre" (a page's pre_texts refused, the contents page included).
  d_is(version, d_tps, members) -> ["OK", hex] or ["REFUSED", reason]
    D(IS) = digest_fields(["TPC-DIS", "1", version, d_tps, m, title_1,
    h_1, ...]): members the other members of the index's
    instruction-artifact class as [title, h] in index order (the caller's
    order is kept), m their count (0 while TPS alone is in the class).
    Reasons: "version"; "digest" (d_tps not 64 lowercase hexadecimal
    characters); "members" (not a list of [non-empty string, 64 lowercase
    hexadecimal characters], or a title twice).

WHAT IS FIXED HERE (the proposal)
  ALG            SHA-256 (agrees with TPS 2.5.4 chain-link hash).
  OUTPUT         lowercase hex, 64 chars.
  ENCODING       a drift-proof, self-delimiting netstring of the ordered
                 fields, so no whitespace, delimiter, or number formatting
                 can drift between residences:
                     enc([b1..bn]) = concat_i( decimal(bytelen(bi)) + ":" + bi )
                 Each field is presented as raw UTF-8 bytes (a string) or as
                 base-10 ASCII (an integer). SHA-256 is taken over enc(...).
  FIELD ORDER    fixed per producer (the caller passes fields in TPS order).
  D(TFK)         combine_digests([D(TPK), D(TrM), D(TMS)]) --- exactly that
                 order (TPS 2.5.3). D(TrM) itself is header-then-entries in
                 fixed order (TPS 2.5.9 / 2.3.10); that per-zone field list
                 is produced by the zone serializer, still owed with the
                 zone schemas (2.2/2.3/2.4) and NOT fixed in this increment.

WHY NETSTRING. A hash chain fails on serialization drift. A self-delimiting
length-prefixed encoding removes every drift source JSON leaves open
(key order, insignificant whitespace, float/number formatting, escaping):
there is exactly one byte sequence for a given ordered field list, and it
is identical in Python and PHP.
"""

import hashlib
import re

TPC_DIGEST_VERSION = "0.3.0"
TPC_DIGEST_STATUS = ("RATIFIED (D97.1, 2026-09-22); values VERIFIED; 0.2.0 declared with TPC-PY 0.2.0 "
                     "(D203.1); 0.3.0 proposed (D216.1)")


def _enc(byte_items):
    """The netstring encoding of an ordered list of byte-strings."""
    out = bytearray()
    for b in byte_items:
        out += str(len(b)).encode("ascii")
        out += b":"
        out += b
    return bytes(out)


def _as_bytes(field):
    """A field's canonical bytes: str -> UTF-8; int (not bool) -> base-10 ASCII."""
    if isinstance(field, bool):
        raise TypeError("digest field must not be bool; use an explicit token")
    if isinstance(field, str):
        return field.encode("utf-8")
    if isinstance(field, int):
        return str(field).encode("ascii")
    if isinstance(field, (bytes, bytearray)):
        return bytes(field)
    raise TypeError("digest field: unsupported type %r" % type(field))


def preimage_fields(fields):
    """The exact bytes hashed by digest_fields --- the auditable artifact.
    Returned as bytes; for ASCII content it prints as the netstring."""
    return _enc([_as_bytes(f) for f in fields])


def digest_fields(fields):
    """SHA-256 over the netstring of the ordered fields; lowercase hex.
    Use for a record's content digest and for each zone once its field
    order is fixed (the caller supplies fields in TPS order)."""
    return hashlib.sha256(preimage_fields(fields)).hexdigest()


def preimage_digests(hex_digests):
    """The exact bytes hashed by combine_digests (auditable)."""
    for h in hex_digests:
        if not _is_hex64(h):
            raise ValueError("combine: each input must be a 64-char lowercase hex digest")
    return _enc([h.encode("ascii") for h in hex_digests])


def combine_digests(hex_digests):
    """SHA-256 over the netstring of an ordered list of hex digests; the
    fixed-order, fixed-algorithm combination rule (TPS 2.5.3 C-02)."""
    return hashlib.sha256(preimage_digests(hex_digests)).hexdigest()


def d_tfk(d_tpk, d_trm, d_tms):
    """D(TFK) = combine_digests([D(TPK), D(TrM), D(TMS)]) --- TPS 2.5.3.
    The container binds to D(TFK)."""
    return combine_digests([d_tpk, d_trm, d_tms])


def _is_hex64(s):
    if not isinstance(s, str) or len(s) != 64:
        return False
    return all(c in "0123456789abcdef" for c in s)


# --------------------------------------------------------------------------
# 0.2.0 (D203.1): the TPS digest D(TPS) and the instruction set digest D(IS)
# --------------------------------------------------------------------------
TPS_ROOT = "The Principia Machina/TPS"     # the tree's root title (record D34.1)
TPS_CONTENTS = TPS_ROOT + "/Contents"
TPS_TAG, TPS_FORM = "TPC-TPS", "1"
DIS_TAG, DIS_FORM = "TPC-DIS", "1"
_WS = " \t\n\x0b\x0c\r"
_VERSION = re.compile(r"(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)")
_SECTION = re.compile(r"((?:0|[1-9][0-9]{0,3})(?:\.(?:0|[1-9][0-9]{0,3}))+) ([^/ ](?:[^/]*[^/ ])?)")
_HEX64 = re.compile(r"[0-9a-f]{64}")


def _refused(reason):
    return ["REFUSED", reason]


def _has_utf8(s):
    try:
        s.encode("utf-8")
        return True
    except UnicodeEncodeError:
        return False


def _ascii_ieq(s, word):
    """s equals word (lower-case ASCII) with ASCII letters in any case."""
    if len(s) != len(word):
        return False
    for a, b in zip(s, word):
        if a != b and a != b.upper():
            return False
    return True


def _pre_tag(t, i):
    """At t[i] == "<": "open" or "close" if a pre tag starts there, else None."""
    for name, kind in (("pre", "open"), ("/pre", "close")):
        j = i + 1 + len(name)
        if _ascii_ieq(t[i + 1:j], name) and (j == len(t) or t[j] in ">/" + _WS):
            return kind
    return None


def pre_texts(wikitext):
    if type(wikitext) is not str or not _has_utf8(wikitext):
        return _refused("pre")
    blocks, start = [], None
    i = wikitext.find("<")
    while i >= 0:
        kind = _pre_tag(wikitext, i)
        if kind == "open":
            if start is not None or not wikitext.startswith("<pre>\n", i):
                return _refused("pre")
            start = i + 6
            i = wikitext.find("<", start)
            continue
        if kind == "close":
            if start is None or not wikitext.startswith("</pre>", i):
                return _refused("pre")
            blocks.append(wikitext[start:i])
            start = None
            i = wikitext.find("<", i + 6)
            continue
        i = wikitext.find("<", i + 1)
    if start is not None:
        return _refused("pre")
    return ["OK", blocks]


def _version_ok(v):
    return type(v) is str and _VERSION.fullmatch(v) is not None


def _title_class(title):
    """("root",), ("contents",), ("section", key) or None."""
    if not _has_utf8(title) or "_" in title or any(ord(c) < 0x20 or ord(c) == 0x7F for c in title):
        return None
    if title == TPS_ROOT:
        return ("root",)
    if title == TPS_CONTENTS:
        return ("contents",)
    if not title.startswith(TPS_ROOT + "/"):
        return None
    m = _SECTION.fullmatch(title[len(TPS_ROOT) + 1:])
    if m is None or "  " in m.group(2):
        return None
    return ("section", tuple(int(p) for p in m.group(1).split(".")))


def tps_pages(pages):
    """0.3.0 (D216.1): the ordered list [title, h] of the kept pages."""
    if type(pages) is not list or any(
            type(p) is not list or len(p) != 2 or type(p[0]) is not str or type(p[1]) is not str
            for p in pages):
        return _refused("pages")
    titles, keys, classes = set(), set(), []
    for title, _ in pages:
        c = _title_class(title)
        if c is None or title in titles or (c[0] == "section" and c[1] in keys):
            return _refused("title")
        titles.add(title)
        if c[0] == "section":
            keys.add(c[1])
        classes.append(c)
    if TPS_ROOT not in titles:
        return _refused("root")
    rows = []
    for (title, text), c in zip(pages, classes):
        r = pre_texts(text)
        if r[0] != "OK":
            return _refused("pre")
        if c[0] == "contents" or not r[1]:
            continue
        h = hashlib.sha256("\n".join(r[1]).encode("utf-8")).hexdigest()
        rows.append(((0,) if c[0] == "root" else (1, c[1]), title, h))
    rows.sort()
    return ["OK", [[title, h] for _, title, h in rows]]


def d_tps(version, pages):
    if not _version_ok(version):
        return _refused("version")
    r = tps_pages(pages)
    if r[0] != "OK":
        return r
    fields = [TPS_TAG, TPS_FORM, version, len(r[1])]
    for title, h in r[1]:
        fields += [title, h]
    return ["OK", digest_fields(fields)]


def d_is(version, d_tps, members):
    if not _version_ok(version):
        return _refused("version")
    if type(d_tps) is not str or _HEX64.fullmatch(d_tps) is None:
        return _refused("digest")
    if type(members) is not list:
        return _refused("members")
    seen = set()
    for m in members:
        if (type(m) is not list or len(m) != 2 or type(m[0]) is not str or not m[0]
                or not _has_utf8(m[0]) or m[0] in seen
                or type(m[1]) is not str or _HEX64.fullmatch(m[1]) is None):
            return _refused("members")
        seen.add(m[0])
    fields = [DIS_TAG, DIS_FORM, version, d_tps, len(members)]
    for title, h in members:
        fields += [title, h]
    return ["OK", digest_fields(fields)]

# END OF ARTIFACT --- tpc_digest 0.3.0
# If this line is absent, the payload arrived truncated.