The tools of the Archive: one definition of normalized text, a literalness check, and the importers

.github/workflows/pr.yml

@@ -64,6 +64,15 @@ jobs:
       - name: Check content for dangerous raw HTML
         run: bin/check-html-safety
 
+      - name: Check the text normalizer
+        run: python3 bin/canon.py --selftest
+
+      - name: Check the archive extraction fixtures
+        # Reads only tests/extraction/, so it needs no Meetup export. It proves
+        # the check still rejects a run that changed an accent, which is the
+        # one defect no normalizer can forgive.
+        run: bin/check-extraction --selftest
+
   links:
     name: Check the links
     runs-on: ubuntu-latest

AGENTS.md

@@ -38,6 +38,32 @@ and checking the output, not by reading the file.
 **`merge.ff = only`.** Any local merge needs `--no-ff`. A failed merge here reports
 "Not possible to fast-forward", which is not a conflict.
 
+**The archive imports run in one order and only once.** `bin/import-meetups`,
+then `bin/import-talks`, then `bin/import-videos`. `bin/import-talks` replaces
+the whole `talks` block of a page and everything after it in the front matter,
+so running it after the others throws away what they added: a re-run over
+`content/events/meetups/` deletes every video link of the talks and every
+`session_videos` block, and puts the speaker credits a human resolved by hand
+back in `speaker_names`. Its docstring has the detail, with the figures of the
+day it was measured. Count what is at stake today with:
+
+```sh
+grep -h 'type: video' content/events/meetups/*.md | wc -l
+grep -l '^session_videos:' content/events/meetups/*.md | wc -l
+```
+
+Re-run any import with `--dry-run` first, and with `--out` pointing at a copy,
+never at `content/`.
+
+**The `bin/import-*` scripts are one-time archaeology, not maintenance.** They
+read an export, the pybcn-archive, that is not in this repository, so that a
+published fact can be traced to a source. They ran once, in the order above,
+and that is the whole of their job. Adding an event, a talk, or a video means
+writing the front matter by hand, as `CONTRIBUTING.md` describes, never
+re-running an importer against `content/`, which is the destructive thing the
+paragraph above measures. The export is not part of maintaining this site, and
+a correction to a published page goes in the page.
+
 **`unsafe=true` in `[markup.goldmark.renderer]`.** Raw HTML in front matter
 and page bodies renders unescaped. Two content files still carry it,
 `content/events/pyday_bcn/pyday_bcn_2018.md` and `pyday_bcn_2019.md`, which
@@ -151,7 +177,7 @@ plausible but different result.
 | `themes/pybcn_theme/layouts/partials/` | Most of the interesting template logic |
 | `static/archives/` | Verbatim mirrors of pre-Hugo sites, copied unprocessed |
 | `data/` | Tables the templates read: `roles.yaml` maps a heading to a role, `role_emoji.yaml` a role to a glyph |
-| `bin/` | Setup and build scripts, plus the gitignored Hugo binary |
+| `bin/` | Setup and build scripts, the checks, `bin/list-channel` (names a channel video the inventory lacks; needs `yt-dlp` and the network), the one-time `bin/import-*` scripts, plus the gitignored Hugo binary |
 | `.hugo-version` | The pinned Hugo version, single source of truth |
 
 The checks, which are what a pull request has to pass:

CONTRIBUTING.md

@@ -122,12 +122,14 @@ Nothing is pushed to `edition` directly.
    verifies its checksum. You do not need Python, Go, or npm for this step.
 3. Run `bin/serve` to see the site at `http://localhost:1313` while you work.
 4. Make the change.
-5. Run the three checks locally, so you find what the pull request would find:
+5. Run the five checks locally, so you find what the pull request would find:
 
    ```
    pip install pyyaml pillow
    bin/check-content
    bin/check-html-safety
+   python3 bin/canon.py --selftest
+   bin/check-extraction --selftest
    bin/hugo --minify -D -d public && bin/check-rendered
    ```
 
@@ -151,6 +153,8 @@ What each check is for:
 | `bin/check-content` | Front matter does not parse, a person `id` does not match its filename, an id is duplicated, a declared photo is missing, an event references a person or sponsor that does not exist, a social URL has the wrong shape, or a field is empty. It warns, without failing, about a photo that is too small or has no colour |
 | `bin/check-html-safety` | Content carries raw HTML that turns a content change into script execution, a redirect, a credential prompt, or a page overlay. See [Content safety check](README.md#content-safety-check) |
 | `bin/check-rendered` | The built pages carry that same markup, which catches a template that produces it even when no content file does, or they link to anything over `http`. See [Rendered page check](README.md#rendered-page-check) |
+| `bin/canon.py --selftest` | The one normalization of imported text stops being idempotent, or changes what it strips |
+| `bin/check-extraction --selftest` | The recorded model runs of `tests/extraction/` stop giving the verdict they are recorded with, one of which has to be a failure |
 | Check the links | Reported, never blocking, because external sites rate-limit |
 
 A merge to `edition` deploys the site. The `github-pages` workflow builds it and

LICENSING.md

@@ -18,8 +18,7 @@ the site:
 - `layouts/`: the Hugo templates and shortcodes of the site.
 - `themes/pybcn_theme/`: the theme. Its templates and partials (`layouts/`),
   its stylesheets (`assets/scss/`), the scripts written for this site
-  (`assets/js/`, except `cookieconsent.min.js`, which is a third-party
-  library, see below), its `archetypes/`, `theme.toml`, and `README.md`. The
+  (`assets/js/`), its `archetypes/`, `theme.toml`, and `README.md`. The
   images and the vendored libraries under the theme are listed under
   [Not licensed](#not-licensed).
 - `bin/`: the build, check, and maintenance scripts.
@@ -155,9 +154,6 @@ with it:
 - `themes/pybcn_theme/static/vendor/font-awesome/webfonts/`: the Font Awesome
   fonts (SIL OFL 1.1).
 - `themes/pybcn_theme/assets/css/bootstrap.min.css`: Bootstrap 4.0.0 (MIT).
-- `themes/pybcn_theme/assets/css/cookieconsent.min.css` and
-  `themes/pybcn_theme/assets/js/cookieconsent.min.js`: the Cookie Consent
-  library by Osano (MIT).
 - `themes/pybcn_theme/assets/images/anon_member.png`: the Font Awesome `user`
   icon as a PNG (CC BY 4.0).
 

README.md

@@ -434,12 +434,17 @@ it. The cost is that the reasons are scattered, so this is the index.
 | MIT for the code, CC BY-SA 4.0 for the content, and nothing for the photographs | [LICENSING.md](LICENSING.md) |
 | `edition` is the source branch and `master` is the built site | [AGENTS.md](AGENTS.md) |
 | Raw HTML in content is gated by three checks rather than banned | [CONTRIBUTING.md](CONTRIBUTING.md) and `bin/check-html-safety` |
+| Imported text is normalized in code, and the model that extracts it is told nothing about typography | `bin/canon.py` and `bin/prompts/extract-talks.md` |
+| An extraction is checked against the normalized source, and one recorded run has to fail | `bin/check-extraction` and [tests/extraction/README.md](tests/extraction/README.md) |
+| A photo PyBCN retweeted is never imported, because its copyright is somebody else's | `bin/import-x-photos` and [LICENSING.md](LICENSING.md) |
+| An import stages its result for review and never writes `content/` itself | `bin/import-x-photos` |
 
 One decision has no file of its own, because it is a decision not to act:
 
-**The heavy image blobs stay in the Git history.** The repository is 139 MB,
-and the largest objects in it are photographs that were replaced long ago,
-up to 5.9 MB each. They are all reachable from `edition` and none from
+**The heavy image blobs stay in the Git history.** The repository is many
+times the size of the site it builds (`git count-objects -vH` gives the
+figure), and the largest objects in it are photographs of several megabytes
+each that were replaced long ago by far smaller files. They are all reachable from `edition` and none from
 `master`, which the deploy workflow force-pushes as a single orphan commit.
 Removing them means `git filter-repo` and a force-push, which changes every
 commit hash from the first touched blob onwards: every clone breaks, every