diff --git a/.github/workflows/pr.yml b/.github/workflows/pr.yml index 019cdbdd8..6d175291b 100644 --- a/.github/workflows/pr.yml +++ b/.github/workflows/pr.yml @@ -64,6 +64,15 @@ jobs: - name: Check content for dangerous raw HTML run: bin/check-html-safety + - name: Check the text normalizer + run: python3 bin/canon.py --selftest + + - name: Check the archive extraction fixtures + # Reads only tests/extraction/, so it needs no Meetup export. It proves + # the check still rejects a run that changed an accent, which is the + # one defect no normalizer can forgive. + run: bin/check-extraction --selftest + links: name: Check the links runs-on: ubuntu-latest diff --git a/AGENTS.md b/AGENTS.md index 943307efd..6ad0e87a0 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -38,6 +38,32 @@ and checking the output, not by reading the file. **`merge.ff = only`.** Any local merge needs `--no-ff`. A failed merge here reports "Not possible to fast-forward", which is not a conflict. +**The archive imports run in one order and only once.** `bin/import-meetups`, +then `bin/import-talks`, then `bin/import-videos`. `bin/import-talks` replaces +the whole `talks` block of a page and everything after it in the front matter, +so running it after the others throws away what they added: a re-run over +`content/events/meetups/` deletes every video link of the talks and every +`session_videos` block, and puts the speaker credits a human resolved by hand +back in `speaker_names`. Its docstring has the detail, with the figures of the +day it was measured. Count what is at stake today with: + +```sh +grep -h 'type: video' content/events/meetups/*.md | wc -l +grep -l '^session_videos:' content/events/meetups/*.md | wc -l +``` + +Re-run any import with `--dry-run` first, and with `--out` pointing at a copy, +never at `content/`. + +**The `bin/import-*` scripts are one-time archaeology, not maintenance.** They +read an export, the pybcn-archive, that is not in this repository, so that a +published fact can be traced to a source. They ran once, in the order above, +and that is the whole of their job. Adding an event, a talk, or a video means +writing the front matter by hand, as `CONTRIBUTING.md` describes, never +re-running an importer against `content/`, which is the destructive thing the +paragraph above measures. The export is not part of maintaining this site, and +a correction to a published page goes in the page. + **`unsafe=true` in `[markup.goldmark.renderer]`.** Raw HTML in front matter and page bodies renders unescaped. Two content files still carry it, `content/events/pyday_bcn/pyday_bcn_2018.md` and `pyday_bcn_2019.md`, which @@ -151,7 +177,7 @@ plausible but different result. | `themes/pybcn_theme/layouts/partials/` | Most of the interesting template logic | | `static/archives/` | Verbatim mirrors of pre-Hugo sites, copied unprocessed | | `data/` | Tables the templates read: `roles.yaml` maps a heading to a role, `role_emoji.yaml` a role to a glyph | -| `bin/` | Setup and build scripts, plus the gitignored Hugo binary | +| `bin/` | Setup and build scripts, the checks, `bin/list-channel` (names a channel video the inventory lacks; needs `yt-dlp` and the network), the one-time `bin/import-*` scripts, plus the gitignored Hugo binary | | `.hugo-version` | The pinned Hugo version, single source of truth | The checks, which are what a pull request has to pass: diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index e6b672f48..7eb00b302 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -122,12 +122,14 @@ Nothing is pushed to `edition` directly. verifies its checksum. You do not need Python, Go, or npm for this step. 3. Run `bin/serve` to see the site at `http://localhost:1313` while you work. 4. Make the change. -5. Run the three checks locally, so you find what the pull request would find: +5. Run the five checks locally, so you find what the pull request would find: ``` pip install pyyaml pillow bin/check-content bin/check-html-safety + python3 bin/canon.py --selftest + bin/check-extraction --selftest bin/hugo --minify -D -d public && bin/check-rendered ``` @@ -151,6 +153,8 @@ What each check is for: | `bin/check-content` | Front matter does not parse, a person `id` does not match its filename, an id is duplicated, a declared photo is missing, an event references a person or sponsor that does not exist, a social URL has the wrong shape, or a field is empty. It warns, without failing, about a photo that is too small or has no colour | | `bin/check-html-safety` | Content carries raw HTML that turns a content change into script execution, a redirect, a credential prompt, or a page overlay. See [Content safety check](README.md#content-safety-check) | | `bin/check-rendered` | The built pages carry that same markup, which catches a template that produces it even when no content file does, or they link to anything over `http`. See [Rendered page check](README.md#rendered-page-check) | +| `bin/canon.py --selftest` | The one normalization of imported text stops being idempotent, or changes what it strips | +| `bin/check-extraction --selftest` | The recorded model runs of `tests/extraction/` stop giving the verdict they are recorded with, one of which has to be a failure | | Check the links | Reported, never blocking, because external sites rate-limit | A merge to `edition` deploys the site. The `github-pages` workflow builds it and diff --git a/LICENSING.md b/LICENSING.md index 2739d6481..c2e1098a5 100644 --- a/LICENSING.md +++ b/LICENSING.md @@ -18,8 +18,7 @@ the site: - `layouts/`: the Hugo templates and shortcodes of the site. - `themes/pybcn_theme/`: the theme. Its templates and partials (`layouts/`), its stylesheets (`assets/scss/`), the scripts written for this site - (`assets/js/`, except `cookieconsent.min.js`, which is a third-party - library, see below), its `archetypes/`, `theme.toml`, and `README.md`. The + (`assets/js/`), its `archetypes/`, `theme.toml`, and `README.md`. The images and the vendored libraries under the theme are listed under [Not licensed](#not-licensed). - `bin/`: the build, check, and maintenance scripts. @@ -155,9 +154,6 @@ with it: - `themes/pybcn_theme/static/vendor/font-awesome/webfonts/`: the Font Awesome fonts (SIL OFL 1.1). - `themes/pybcn_theme/assets/css/bootstrap.min.css`: Bootstrap 4.0.0 (MIT). -- `themes/pybcn_theme/assets/css/cookieconsent.min.css` and - `themes/pybcn_theme/assets/js/cookieconsent.min.js`: the Cookie Consent - library by Osano (MIT). - `themes/pybcn_theme/assets/images/anon_member.png`: the Font Awesome `user` icon as a PNG (CC BY 4.0). diff --git a/README.md b/README.md index 909dc6e0c..76ae96335 100644 --- a/README.md +++ b/README.md @@ -434,12 +434,17 @@ it. The cost is that the reasons are scattered, so this is the index. | MIT for the code, CC BY-SA 4.0 for the content, and nothing for the photographs | [LICENSING.md](LICENSING.md) | | `edition` is the source branch and `master` is the built site | [AGENTS.md](AGENTS.md) | | Raw HTML in content is gated by three checks rather than banned | [CONTRIBUTING.md](CONTRIBUTING.md) and `bin/check-html-safety` | +| Imported text is normalized in code, and the model that extracts it is told nothing about typography | `bin/canon.py` and `bin/prompts/extract-talks.md` | +| An extraction is checked against the normalized source, and one recorded run has to fail | `bin/check-extraction` and [tests/extraction/README.md](tests/extraction/README.md) | +| A photo PyBCN retweeted is never imported, because its copyright is somebody else's | `bin/import-x-photos` and [LICENSING.md](LICENSING.md) | +| An import stages its result for review and never writes `content/` itself | `bin/import-x-photos` | One decision has no file of its own, because it is a decision not to act: -**The heavy image blobs stay in the Git history.** The repository is 139 MB, -and the largest objects in it are photographs that were replaced long ago, -up to 5.9 MB each. They are all reachable from `edition` and none from +**The heavy image blobs stay in the Git history.** The repository is many +times the size of the site it builds (`git count-objects -vH` gives the +figure), and the largest objects in it are photographs of several megabytes +each that were replaced long ago by far smaller files. They are all reachable from `edition` and none from `master`, which the deploy workflow force-pushes as a single orphan commit. Removing them means `git filter-repo` and a force-push, which changes every commit hash from the first touched blob onwards: every clone breaks, every diff --git a/bin/canon.py b/bin/canon.py new file mode 100644 index 000000000..b2b4b169f --- /dev/null +++ b/bin/canon.py @@ -0,0 +1,286 @@ +"""Canonical text form for the Meetup archive import. + +The decision this module encodes: text imported from the Meetup export is +stored normalized and consistent, not as a facsimile of Meetup. The +normalization has to be deterministic and in code, with one definition, and +it cannot depend on a model behaving well. A model told NOT to normalize +the typography was measured against the Meetup export: it changed the +apostrophe anyway, and in the same pass it corrupted an accented letter +that it had copied correctly without the rule. Code cannot do that, +because it never touches a letter. + +canon() is the normalizer. strip_md() and prep() support comparing a source +string against an extracted one: the extractor runs text through Markdown, +and a source string never was, so the two cannot be compared until the +Markdown is gone from both. strip_time() removes a schedule label's leading +time, because the time lives in the event's own start and end fields and +does not belong in the label text too. + +The one rule canon() never breaks: it does not touch a letter or an accent. +Everywhere else, "normalize the typography" is safe to read as "make it +closer to ASCII", and a model treats it that way. Here it is not: the data +covers Catalan, Spanish, and French, so NFKD-then-strip-accents or any other +latinizing transform would silently corrupt "Història" into "Historia" or +"Historia" into something that no longer matches its own source. The map +below stays within the set of marks and spaces that normal keyboard input +also maps to one of these forms; it never reaches into the letters. +""" + +import argparse +import re +import sys +import unicodedata + +# Every substitution below is built from its code point, not typed as the +# raw character. Several of these marks look alike, or look like nothing at +# all (a hair space renders as a gap the width of a typo), and a source file +# that carries them as raw bytes hides that from a reviewer: the same trap +# bin/check-html-safety guards against in built HTML. Spelling out the code +# point also keeps these tables in the ASCII range, so a diff of one never +# carries an invisible or look-alike character by accident. The rest of the +# file is not ASCII, and cannot be: the selftest table below holds a raw +# curly apostrophe and raw accented letters, because those are what it +# tests. + +# Quotation marks and primes that stand in for a straight quote on a +# keyboard that supports typing one directly, or that a word processor +# "smart quotes" feature produces from one. Catalan text uses the left and +# right single forms for its apostrophe (d'una), so both single and double +# families are mapped, not just the double ones a generic cleanup usually +# targets. +QUOTES = { + 0x2018: "'", 0x2019: "'", 0x201A: "'", 0x201B: "'", # single quotes + 0x201C: '"', 0x201D: '"', 0x201E: '"', # double quotes + 0x00AB: '"', 0x00BB: '"', # guillemets, used as quotes in French text + 0x2032: "'", 0x2033: '"', # prime and double prime +} + +# Dashes and the mathematical minus sign that a source pastes in place of a +# plain hyphen: Meetup descriptions carry an en dash, an em dash, and the +# Unicode minus sign interchangeably with "-". All of them collapse to the +# one ASCII hyphen, so a comparison never fails over which dash a human +# picked. +DASHES = { + 0x2013: "-", # en dash + 0x2014: "-", # em dash + 0x2212: "-", # minus sign + 0x2010: "-", # hyphen (the dedicated Unicode character, not "-") + 0x2011: "-", # non-breaking hyphen +} + +# A single ellipsis character, which prep() has to treat as the same text as +# three periods: a source may use either, and the extractor's model tends to +# write the three-period form regardless of which one the source used. +ELLIPSIS = {0x2026: "..."} + +# Non-breaking, figure, narrow, and thin spaces. Meetup's export carries these +# where a human just pressed the space bar on a system that substituted one, +# or where an editor inserted a non-breaking space before a unit or a +# punctuation mark. None of them is a letter, so all of them fold to the +# plain space. +SPACES = { + 0x00A0: " ", # no-break space + 0x2007: " ", # figure space + 0x202F: " ", # narrow no-break space + 0x2009: " ", # thin space + 0x200A: " ", # hair space +} + +_CANON_MAP = {} +for _table in (QUOTES, DASHES, ELLIPSIS, SPACES): + _CANON_MAP.update({chr(codepoint): replacement for codepoint, replacement in _table.items()}) +_CANON_TABLE = str.maketrans(_CANON_MAP) + +_HORIZONTAL_WHITESPACE = re.compile(r"[ \t]+") + + +def canon(text): + """Return the acceptable typographic form of text. + + NFC first, so a letter with a combining accent and the same letter with + a precomposed accent compare equal. Then the character substitutions + above: curly and angle quotes to a straight quote, every dash and the + minus sign to a hyphen, a one-character ellipsis to three periods, and + the non-ASCII spaces to a plain one. Runs of plain spaces and tabs + collapse to one space. + + This never changes a letter or an accent. That boundary is the entire + point of the function: the archive's source text is Catalan, Spanish, and + French, and a transform that also "normalizes" accents would turn + "Història" into "Historia" or "Historia" into "História". Either breaks + a comparison against the source and nobody would notice until a word + read wrong on the published page. + """ + text = unicodedata.normalize("NFC", text or "") + text = text.translate(_CANON_TABLE) + return _HORIZONTAL_WHITESPACE.sub(" ", text) + + +# [text](url) resolves to text. Applied before the character-class strip +# below, because the URL inside the parentheses can itself contain '*', '_', +# or '#', and stripping those first would leave the parentheses and the URL +# behind as stray text. +_MARKDOWN_LINK = re.compile(r"\[([^\]]*)\]\([^)]*\)") + +# Markdown markup characters with no reading-order meaning of their own: +# emphasis (* _), blockquote (>), inline code (`), and heading (#). Removing +# them, rather than parsing Markdown into a tree, is enough because prep() +# only needs the reading text, not the structure. +_MARKDOWN_MARKUP = re.compile(r"[*_>`#]") + +_WHITESPACE_RUN = re.compile(r"\s+") + + +def strip_md(text): + """Return the Markdown-free reading text of text, for comparison. + + The extractor's output is Markdown; a source string pasted out of the + Meetup export never was. The two cannot be compared for equality until + both go through this: it resolves a link to its visible text, drops the + backslash of an escaped character, drops the emphasis/quote/code/heading + markup characters, drops every emoji and formatting character (Unicode + categories So "other symbol", Sk "modifier symbol", and Cf "format", which + covers variation selectors and zero-width joiners), and collapses all + whitespace, including a newline, to one space. + """ + text = (text or "").replace("\\", "") + text = _MARKDOWN_LINK.sub(r"\1", text) + text = _MARKDOWN_MARKUP.sub("", text) + text = "".join( + character for character in text + if unicodedata.category(character) not in ("So", "Sk", "Cf") + ) + return _WHITESPACE_RUN.sub(" ", text).strip() + + +def prep(text): + """Return text ready to compare: strip_md(canon(text)). + + This is what the extractor and the check run both the source and the + extracted text through before comparing them, so neither side's leftover + typography or Markdown can make a true match look like a mismatch, or + the reverse. + """ + return strip_md(canon(text)) + + +# A grid label carries its own leading time ("18:30: Welcome and Talks") +# because that is how it read on the Meetup page, but the imported event +# already stores the time in its start and end fields. Leaving it in the +# label would duplicate it and risk the two disagreeing after an edit to +# either one. The pattern accepts the two separators the export uses (":" or +# ".") between hour and minute, an optional "h" or "hs" unit after the +# minutes, and a colon, a hyphen, or nothing between the time and the label +# text. It runs after prep(), so by this point any dash has already become +# the plain hyphen "-" that canon() produces; there is no other dash form to +# match here. +# +# "hs" has to come before "h" in the alternation. Regex alternation tries +# each branch in order and stops at the first one that lets the rest of the +# pattern match, not the longest one: with "h" first, "9:00hs: Talk" matched +# only the "h", because the remaining "s: Talk" still satisfied the optional +# groups that follow, and the label came out as "s: Talk" instead of "Talk". +_TIME = r"\d{1,2}[:.]\d{2}\s*(?:hs|h)?\s*" +# The optional second time covers a range ("09:30 - 10:45 Keynote"), which is +# how the PyDay and PyData grids print a slot. Without it only the start time +# came off and the end time stayed in the label text. +_LEADING_TIME = re.compile(r"^" + _TIME + r"(?:[:-]\s*" + _TIME + r")?[:-]?\s*") + + +def strip_time(label): + """Return label with a leading grid time removed, for example: + + "18:30: Welcome and Talks" -> "Welcome and Talks" + "Welcome and Talks" -> "Welcome and Talks" (unchanged, no leading time) + """ + return _LEADING_TIME.sub("", prep(label)) + + +# --------------------------------------------------------------------------- +# Self-test +# --------------------------------------------------------------------------- + +# Each row is one behavior this module has to keep. "kind" says which +# function the row checks, "input" is what goes in, "expect" is what has to +# come out. This is the only test this repository has for canon.py, so the +# table is also where a regression gets added: a bug found later becomes a +# new row here, not a note in a pull request description. +_SELFTEST_CASES = [ + ("canon", "d’una", "d'una"), # Catalan curly apostrophe + ("canon", "Història", "Història"), # accent must not change + ("canon", "", ""), + ("canon", None, ""), + ("strip_md", None, ""), + ("strip_md", + "how [Yaetos](https://github.com/arthurprevot/yaetos) can help", + "how Yaetos can help"), + ("prep", None, ""), + ("strip_time", "18:30: Welcome and Talks", "Welcome and Talks"), + ("strip_time", "18.30 Welcome and Talks", "Welcome and Talks"), + ("strip_time", "9:00h - Talk", "Talk"), + ("strip_time", "9:00hs: Talk", "Talk"), + ("strip_time", "09:30 - 10:45 Keynote", "Keynote"), # a range, as PyDay prints it + ("strip_time", "09:30-10:45 Keynote", "Keynote"), + ("strip_time", "10:00 - 10:30: Coffee break", "Coffee break"), + ("strip_time", "Welcome and Talks", "Welcome and Talks"), # no leading time + ("strip_time", None, ""), +] + +_FUNCTIONS = {"canon": canon, "strip_md": strip_md, "prep": prep, "strip_time": strip_time} + + +def _run_selftest(): + """Run _SELFTEST_CASES plus the two checks that do not fit a (kind, input, + expect) row, print one line per case, and return True if all of them pass. + """ + passed = True + + for kind, value, expect in _SELFTEST_CASES: + got = _FUNCTIONS[kind](value) + ok = got == expect + passed = passed and ok + print("%-4s %-10s %-45r -> %-25r (expect %r)" + % ("ok" if ok else "FAIL", kind, value, got, expect)) + + # Història and História must stay distinct after prep(), because they + # are different words in different languages. A transform that folded + # accents, rather than merely normalizing typography, would make this + # pair equal and silently merge two different source texts. + distinct = prep("Història") != prep("História") + passed = passed and distinct + print("%-4s %-10s %r != %r" + % ("ok" if distinct else "FAIL", "prep", "Història", "História")) + + # canon() has to be idempotent: running it twice must give the same + # result as running it once, because prep() calls it once but the + # extractor and the check may each run it again on the other's output. + for kind, value, _ in _SELFTEST_CASES: + if kind != "canon" and value is None: + continue + once = canon(value) + twice = canon(once) + ok = once == twice + passed = passed and ok + print("%-4s %-10s canon(canon(%r)) == canon(%r)" + % ("ok" if ok else "FAIL", "idempotent", value, value)) + + return passed + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--selftest", action="store_true", + help="run the built-in test table and exit 0 or 1") + args = parser.parse_args() + + if not args.selftest: + parser.print_help() + return 1 + + ok = _run_selftest() + print("\nPASS" if ok else "\nFAIL") + return 0 if ok else 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/bin/check-extraction b/bin/check-extraction new file mode 100755 index 000000000..33892cceb --- /dev/null +++ b/bin/check-extraction @@ -0,0 +1,926 @@ +#!/usr/bin/env python3 +"""Validate a talk extraction against the Meetup source it was read from. + +The archive import cannot parse 19 years of volunteer-written Markdown with +one parser, so a model reads each event description and returns the talks it +announced. A model can rewrite a word instead of copying it, and a rewritten +word on a published page looks like the real one. This check is what makes +that impossible to miss. + +It compares every extracted title and description against the normalized +source, not against the raw source. The extractor's output went through +Markdown and the source string never did, so the raw forms cannot be +compared at all: a link in the source ("[Yaetos](https://...)") makes a +correct abstract look invented. bin/canon.py defines the one normalization +both sides go through, and it never touches a letter or an accent. So a +curly apostrophe turned straight passes, and a changed accent fails. That +boundary is the whole point: the archive holds Catalan, Spanish, and French +text, where "Historia" with an acute accent and with a grave accent are two +different words. + +Three modes: + + Literalness (the default) reads an extraction and its source events, and + reports every field that is not a substring of the normalized source, + with the divergence located and classified: spacing (a warning), + accent_or_case (an error), or content (an error). It also reports a field + the source does not ground, a description the model cut short (an error), + and a shape defect such as a missing event. + + Two divergences from the source are not defects, and the literalness + mode resolves them instead of reporting them. A bare URL in parentheses + may be absent from a title or an abstract, because it is a reference and + not prose: see drop_urls(). A duration may be the span between two times + of the source, because reading "09:30" and "10:30" as 60 minutes is a + derivation: see spans_of(). + + The substring test has one hole by construction: a shortened text is + still a substring of the one it was cut from, so a model that drops half + an abstract passes it. Losing half a description in an archive nobody + reads again is exactly the silent failure this check exists to prevent, + so a description that stops in the middle of its own source line, at no + sentence end, is reported as truncated. See truncation(). + + Benchmark scores an extraction against tests/extraction/truth.yaml, the + reference read by hand before any model ran. + + Selftest runs both modes over the 7 recorded runs in + tests/extraction/model-runs/ and compares the result with + tests/extraction/expectations.yaml, field by field. One recorded run is + corrupted on purpose and has to fail, with exactly one accent_or_case + error. A check that only ever sees good input proves nothing. + +Exit codes: + 0 no error. Warnings do not fail the build + 1 at least one error, or a selftest result that differs from the + expectations + 2 the check could not run: bad arguments, a missing file, or a file it + cannot parse + +Usage: + bin/check-extraction EXTRACTION.json EVENTS.json + bin/check-extraction --benchmark EXTRACTION.json [--truth TRUTH.yaml] + bin/check-extraction --selftest +""" + +import argparse +import difflib +import json +import pathlib +import re +import sys +import unicodedata + +import yaml + +# canon.py sits next to this script and is not an installed package. +sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent)) + +from canon import canon, prep, strip_time # noqa: E402 + +ROOT = pathlib.Path(__file__).resolve().parent.parent +FIXTURES = ROOT / "tests" / "extraction" +MODEL_RUNS = FIXTURES / "model-runs" +EVENTS = FIXTURES / "events.json" +TRUTH = FIXTURES / "truth.yaml" +EXPECTATIONS = FIXTURES / "expectations.yaml" + +# The two text fields the model has to copy and not write. Everything else it +# returns is either a name, a number, or a structure, and each of those has +# its own check below. +TEXT_FIELDS = ("title", "description") + +# A schedule is "agenda" in bin/prompts/extract-talks.md. The recorded runs +# predate that name and write "non_talk_items", so both are read: the check +# has to work on the fixtures it ships with. +AGENDA_KEYS = ("agenda", "non_talk_items") + +# The shortest run of characters that anchors the extracted text to its place +# in the source. Three characters or fewer match by accident ("the", " a "), +# which drags the window to the wrong paragraph and turns one changed letter +# into a whole false divergence. A long field needs a long anchor for the +# same reason, so the floor grows with the field: an anchor has to be at +# least one ANCHOR_SHARE of it. Without that, an invented sentence anchored +# on " a " and "ing" produced a window the length of the whole event and a +# report nobody could read. +MIN_ANCHOR = 4 +ANCHOR_SHARE = 8 + +# How much of the source to print on each side of a divergence, how long a +# single fragment may get before its middle is elided, and how many +# divergences to print for one field. A field that diverges everywhere is +# invented text, and the first few spans already say so. +CONTEXT = 30 +MAX_FRAGMENT = 60 +MAX_SPANS = 3 + +# How much of the field, and of the source after it, a truncation report +# prints. Enough to show the word the cut fell inside and the clause that +# follows it, short enough to keep each line readable. +CONTINUATION = 50 + +# The characters a complete description is allowed to stop at: the sentence +# marks, a closing parenthesis, and the straight quotes canon() produces +# from the curly ones. A description that ends on one of these is finished, +# even when its source line carries more text after it, because the line +# usually goes on with the next field of the template. +SENTENCE_END = ".!?:)'\"" + + +# --------------------------------------------------------------------------- +# Comparison forms +# +# These three reductions classify a divergence. None of them ever runs over +# text this repository stores: fold() strips accents, which canon.py refuses +# to do precisely because it corrupts Catalan. Here that is the point. A pair +# that differs before folding and matches after it is a changed accent, which +# is the defect this check exists to catch. +# --------------------------------------------------------------------------- + +def as_text(value): + """Return value as a string, so a malformed field cannot crash the check. + + An extraction is model output. A field that should hold a string can hold + a number or null, and the check has to report that, not raise on it. + """ + if value is None: + return "" + return value if isinstance(value, str) else str(value) + + +def alnum(text): + """Return only the letters and digits of text, in order.""" + return "".join(character for character in text if character.isalnum()) + + +def fold(text): + """Return text lowercased and without accents, for classification only.""" + decomposed = unicodedata.normalize("NFD", text.lower()) + return "".join(character for character in decomposed + if not unicodedata.combining(character)) + + +def loose(text): + """Return a title reduced for matching against the reference. + + prep() first, then lowercase, then drop everything that is not a letter, + a digit, an underscore, or a space. The underscore survives, because the + character class keeps a word character and an underscore is one. No title + in the archive turns on an underscore, so that costs nothing. + + The reference writes a title as the source does, and an extraction may + keep or drop a trailing period, a colon, or a pair of quotes without + being wrong about which talk it found. + """ + reduced = re.sub(r"[^\w\s]", " ", prep(as_text(text)).lower(), flags=re.UNICODE) + return re.sub(r"\s+", " ", reduced).strip() + + +# --------------------------------------------------------------------------- +# Locating a divergence +# --------------------------------------------------------------------------- + +def source_text(event): + """Return the normalized text an extraction of this event may copy from. + + The event title counts as source. A workshop with no talk title of its + own takes the event's, which bin/prompts/extract-talks.md asks for, so a + check that read only the description would report that title as invented. + """ + title = as_text(event.get("title")) + description = as_text(event.get("description")) + return prep(f"{title}\n{description}") + + +def source_lines(event): + """Return the normalized source of this event, one entry per source line. + + source_text() collapses the newlines, because a field the model copied + may span them and a comparison must not fail over that. Truncation needs + the opposite: the line is what says where a text ends. canon() keeps the + newlines, so the split happens on the normalized text, and prep() then + runs over each line on its own. + """ + title = as_text(event.get("title")) + description = as_text(event.get("description")) + return [prep(line) for line in canon(f"{title}\n{description}").split("\n")] + + +def truncation(lines, extracted): + """Return the source text that follows a cut-short field, or None. + + extracted is already prep()ed, and the caller has established that it + copies the source. The field is cut short when its source line goes on + after it and it stops at no sentence end. A field that ends a line is + complete, whatever its last character is. + + Nothing here looks at the title: a title legitimately ends on a letter, + and the event title sits on the same line as nothing else, so the rule + would report every one of them. The caller applies this to descriptions + only. + """ + if not extracted or extracted[-1] in SENTENCE_END: + return None + rest = None + for line in lines: + start = line.find(extracted) + while start >= 0: + after = line[start + len(extracted):] + # One occurrence that ends its line is enough to say the field is + # whole, even when another copy of the same text sits mid-line. + if not after: + return None + if rest is None: + rest = after + start = line.find(extracted, start + 1) + return rest + + +# The Meetup export prints a link as its visible text followed by the bare +# address in parentheses: "Ansible (http://www.ansible.com)". A model that +# copies the sentence it reads leaves that parenthesis out, and it is right +# to: an address is a reference, not prose, and the model returns it in +# `links`, which is where the import takes it from. This is the decision +# prep() already makes for the other form the same link takes in this +# archive, where "[text](url)" resolves to "text". +# +# Only a parenthesis that holds nothing but an address goes. A parenthesis +# with prose in it is prose, and a changed word inside one still fails. +_BARE_URL = re.compile(r"\s*\((?:https?://|www\.)[^\s()]*\)") + + +def drop_urls(text): + """Return text without the parentheses that hold only a bare URL. + + The substitution repeats until nothing changes. The export nests the + form inside itself when the visible text is an address as well, as in + "Kinto (https://kinto.readthedocs.io (https://kinto.readthedocs.io/))", + and one pass leaves the outer parenthesis behind. + """ + while True: + shorter = _BARE_URL.sub("", text) + if shorter == text: + return text + text = shorter + + +def align(source, extracted): + """Return the slice of source that lines up with extracted, or None. + + The window is anchored on the first and the last long matching run, and + then extended by however much of extracted hangs off each end, so the + window and extracted have the same length when the only difference is a + substitution. None means extracted has too little in common with the + source to align at all, which is a content defect and needs no window. + """ + matcher = difflib.SequenceMatcher(None, source, extracted, autojunk=False) + floor = max(MIN_ANCHOR, len(extracted) // ANCHOR_SHARE) + anchors = [block for block in matcher.get_matching_blocks() if block.size >= floor] + # Half of the field has to sit on an anchor. Below that the field is not + # a copy of anything in the source, and aligning it against the nearest + # paragraph only produces a long report about a text the source never + # had. "Not in the source" is both true and shorter. + if not anchors or sum(anchor.size for anchor in anchors) * 2 < len(extracted): + return None + first, last = anchors[0], anchors[-1] + start = max(0, first.a - first.b) + tail = len(extracted) - (last.b + last.size) + end = min(len(source), last.a + last.size + max(tail, 0)) + return start, end + + +def divergences(window, extracted): + """Return (core, spans) for a window and the text aligned against it. + + core is the window with its border noise removed. A window cut out of a + longer source carries a head and a tail that belong to the source and not + to the extracted field: the "Abstract: " that sits right before the text + is the one that bit us. Those show up as a deletion that touches an edge + of the window and consumes none of the extracted text, and they are not + divergences. Everything else is, and comes back as a span of + (source fragment, extracted fragment) with its offset in core. + """ + matcher = difflib.SequenceMatcher(None, window, extracted, autojunk=False) + opcodes = matcher.get_opcodes() + + head, tail = 0, len(window) + kept = [] + for index, (tag, i1, i2, j1, j2) in enumerate(opcodes): + if tag == "equal": + continue + border = (index == 0 and i1 == 0 and j1 == 0) or ( + index == len(opcodes) - 1 and i2 == len(window) and j2 == len(extracted)) + if tag == "delete" and border: + if i1 == 0: + head = i2 + else: + tail = i1 + continue + kept.append((i1, i2, j1, j2)) + + core = window[head:tail] + spans = [(i1 - head, window[i1:i2], extracted[j1:j2]) for i1, i2, j1, j2 in kept] + return core, spans + + +def classify(core, extracted): + """Return the class of the difference between core and extracted. + + spacing the letters and digits are the same, so only space or + punctuation moved. A warning: canon.py already decides + the typography of what gets stored. + accent_or_case the same text once accents and case are folded away, and + not before. An error, and the one this check is for. + content anything else: another word, another number, or text the + source does not have. An error. + """ + if alnum(core) == alnum(extracted): + return "spacing" + if fold(core) == fold(extracted) or fold(alnum(core)) == fold(alnum(extracted)): + return "accent_or_case" + return "content" + + +def show(core, spans): + """Return one (source line, extracted line) pair per divergence. + + Each line is the fragment in its context, with the divergence itself in + square brackets. A failure a human cannot read is worth nothing, and the + change is usually one letter in a 500-character abstract. + """ + lines = [] + for offset, from_source, from_extracted in spans[:MAX_SPANS]: + before = core[max(0, offset - CONTEXT):offset] + after = core[offset + len(from_source):offset + len(from_source) + CONTEXT] + lead = "..." if offset > CONTEXT else "" + trail = "..." if offset + len(from_source) + CONTEXT < len(core) else "" + lines.append(( + f"{lead}{before}[{elide(from_source)}]{after}{trail}", + f"{lead}{before}[{elide(from_extracted)}]{after}{trail}", + )) + left = len(spans) - len(lines) + if left: + lines.append((f"({left} more divergences in this field)", "")) + return lines + + +def elide(fragment): + """Return fragment with its middle cut out once it gets too long.""" + if len(fragment) <= MAX_FRAGMENT: + return fragment + half = (MAX_FRAGMENT - 5) // 2 + return f"{fragment[:half]} ... {fragment[-half:]}" + + +# --------------------------------------------------------------------------- +# Mode 1: literalness +# --------------------------------------------------------------------------- + +def check_field(event_id, where, field, value, source, lines): + """Return the findings for one text field. An empty list means it copies. + + A finding is a dict with the event, the field, the class, and the aligned + fragments to print. + + A field that is a substring of the source copies it, which leaves only + one question: whether it copies all of it. lines carries the source + split by line, and truncation() answers that for a description. + """ + extracted = prep(as_text(value)) + if not extracted: + return [] + + # The field is read twice: as it stands, and with every bare + # parenthesized URL dropped from both sides. Both sides, because one + # abstract keeps one of those references and drops the next. The second + # form is only built when there is a reference to drop, so a field that + # carries none is compared exactly once, as before. + forms = [(extracted, source, lines)] + without_urls = drop_urls(extracted) + if without_urls != extracted or drop_urls(source) != source: + forms.append((without_urls, drop_urls(source), + [drop_urls(line) for line in lines])) + + for copied, within, inside in forms: + if copied not in within: + continue + rest = truncation(inside, copied) if field == "description" else None + if rest is None: + return [] + return [{ + "event_id": event_id, "where": where, "field": field, + "klass": "truncated", + "lines": [ + (f"the field stops at: {copied[-CONTINUATION:]!r}", ""), + (f"the source line goes on: {rest[:CONTINUATION]!r}", ""), + ], + }] + + located = align(source, extracted) + if located is None: + return [{ + "event_id": event_id, "where": where, "field": field, + "klass": "content", "lines": [("(nothing in the source)", extracted)], + }] + + start, end = located + core, spans = divergences(source[start:end], extracted) + if not spans: + # The window carried only border noise, so the field does copy the + # source and the substring test failed on the edges alone. + return [] + return [{ + "event_id": event_id, "where": where, "field": field, + "klass": classify(core, extracted), "lines": show(core, spans), + }] + + +# A clock reading in the source, written with either separator the export +# uses: "18:30" and "18.30" are both times, which bin/canon.py already +# documents for the schedule labels it strips a time from. +# +# The hour stops at 23 and the minute at 59, so a figure that cannot be a +# time is not read as one. "Python 3.5" still reads as 3:05, and no pattern +# tells those two apart. That costs nothing here: a duration is accepted +# only when it equals the span between two readings, so a stray one has to +# land on the exact number the extraction claims. +_CLOCK = re.compile(r"(?6}{'required':>10}{'missing':>9}" \ + f"{'spurious':>10}{'speakers':>10}{'ambiguous':>11}" + print(header) + print("-" * len(header)) + for row in score["per_event"]: + print(f"{row['event_id']:<12}{row['talks']:>6}{row['required']:>10}" + f"{row['required_missing']:>9}{row['spurious']:>10}" + f"{row['wrong_speakers']:>10}{row['ambiguous_included']:>11}") + print("-" * len(header)) + print(f"{'total':<12}{score['talks']:>6}{'':>10}{score['required_missing']:>9}" + f"{score['spurious']:>10}{score['wrong_speakers']:>10}" + f"{score['ambiguous_included']:>11}") + + +# --------------------------------------------------------------------------- +# Mode 3: the selftest +# --------------------------------------------------------------------------- + +# The keys expectations.yaml records, and where each one comes from. Both +# sections read the same measurement: ungrounded_fields is measured by the +# literalness check and recorded under benchmark. +LITERAL_KEYS = ("verdict", "fields_checked", "identical_to_normalized_source", + "spacing_warnings", "accent_or_case_errors", "content_errors", + "truncated_errors") +BENCHMARK_KEYS = ("talks", "required_missing", "spurious", "wrong_speakers", + "ungrounded_fields", "ambiguous_included") + + +def run_selftest(): + """Measure every recorded run and compare it with the expectations. + + Return the number of runs that do not match. The fixtures are the only + regression test this check has, so a mismatch is a defect in the check: + expectations.yaml and truth.yaml are the reference, never the thing to + edit. + """ + events = load_json(EVENTS) + truth = load_yaml(TRUTH) + expectations = load_yaml(EXPECTATIONS) + + failed = 0 + for name in sorted(expectations): + expected = expectations[name] + path = MODEL_RUNS / f"{name}.json" + if not path.exists(): + print(f"FAIL {name}: {path} is missing") + failed += 1 + continue + + extraction = load_json(path) + literal = run_literal(extraction, events) + benchmark = run_benchmark(extraction, truth) + benchmark["ungrounded_fields"] = literal["ungrounded_fields"] + + got = { + "literal": {key: literal[key] for key in LITERAL_KEYS}, + "benchmark": {key: benchmark[key] for key in BENCHMARK_KEYS}, + } + + differences = [] + for section in ("literal", "benchmark"): + for key, want in (expected.get(section) or {}).items(): + mine = got[section].get(key) + if mine != want: + differences.append(f"{section}.{key}: expected {want!r}, got {mine!r}") + + if differences: + failed += 1 + print(f"FAIL {name}") + for line in differences: + print(f" {line}") + for finding in literal["findings"]: + print(f" measured: event {finding['event_id']} " + f"{finding['field']} {finding['klass']}") + else: + print(f"ok {name}: {expected['literal']['verdict']}, " + f"{got['literal']['fields_checked']} fields, " + f"{got['literal']['accent_or_case_errors']} accent_or_case, " + f"{got['literal']['content_errors']} content, " + f"{got['benchmark']['talks']} talks, " + f"{got['benchmark']['required_missing']} missing, " + f"{got['benchmark']['spurious']} spurious") + + total = len(expectations) + print(f"\n{total - failed} of {total} recorded runs match the expectations.") + return failed + + +# --------------------------------------------------------------------------- +# Entry point +# --------------------------------------------------------------------------- + +def unreadable(path, problem): + """Leave with code 2: the check could not run, and found no defect.""" + print(f"cannot read {path}: {problem}", file=sys.stderr) + sys.exit(2) + + +def load_json(path): + try: + with open(path, encoding="utf-8") as handle: + return json.load(handle) + except (OSError, ValueError) as problem: + unreadable(path, problem) + + +def load_yaml(path): + try: + with open(path, encoding="utf-8") as handle: + return yaml.safe_load(handle) or {} + except (OSError, yaml.YAMLError) as problem: + unreadable(path, problem) + + +def main(): + parser = argparse.ArgumentParser( + description="Validate a talk extraction against its normalized source.") + parser.add_argument("extraction", nargs="?", help="the extraction JSON") + parser.add_argument("events", nargs="?", help="the source events JSON") + parser.add_argument("--benchmark", action="store_true", + help="score the extraction against truth.yaml") + parser.add_argument("--truth", default=str(TRUTH), + help=f"the reference to score against (default {TRUTH})") + parser.add_argument("--selftest", action="store_true", + help="measure the recorded runs against expectations.yaml") + args = parser.parse_args() + + if args.selftest: + if args.extraction or args.benchmark: + parser.error("--selftest takes no other argument") + return 1 if run_selftest() else 0 + + if args.benchmark: + if not args.extraction or args.events: + parser.error("--benchmark takes one extraction file") + score = run_benchmark(load_json(args.extraction), load_yaml(args.truth)) + report_benchmark(score) + # A missing required talk, a spurious one, or a wrong speaker is an + # error, so this mode exits 1 like the other one. An ambiguous talk + # never counts, in either direction: the source does not settle it. + return 1 if (score["required_missing"] or score["spurious"] + or score["wrong_speakers"]) else 0 + + if not args.extraction or not args.events: + parser.error("give an extraction file and an events file") + result = run_literal(load_json(args.extraction), load_json(args.events)) + return 1 if report_literal(result) else 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/bin/check-html-safety b/bin/check-html-safety index f94680469..8060333ba 100755 --- a/bin/check-html-safety +++ b/bin/check-html-safety @@ -55,6 +55,13 @@ TEXT_SUFFIXES = ( ".pandoc", ".pdc", # text/pandoc ".txt", # text/plain, copied as a page resource ".svg", # image/svg+xml, XML that can carry