Repository navigation
The tools of the Archive: one definition of normalized text, a literalness check, and the importers - #214
Merged
Conversation
…atch Three modules under bin/, one rule each, so that the import scripts that follow cannot each carry a copy that drifts. bin/canon.py normalizes text read from the Meetup export: one deterministic transform in code, which never touches a letter or an accent. A model told not to change the typography was measured against the export: it changed the apostrophe anyway and corrupted an accented letter in the same pass, so the rule cannot live in a prompt. The module also strips Markdown and a leading time from a schedule label, which is what comparing a source string against an extracted one needs. It has a --selftest. bin/slug.py makes an address out of a title: the file name of a meetup and the id of a talk, which is the last segment of the URL of its page. It drops the accents, which canon() must never do to a stored string. bin/titlematch.py recognises a title inside another string that carries it: the caption of a photo that quotes the talk it shows, or the title of a video upload that names the talk it recorded. The two titles are two honest records of one session at two moments, so the rule is tolerant and documented, and it lives here once.
…fixtures The import cannot parse 19 years of volunteer-written Markdown with one parser, so a model reads each event description and returns the talks it announced. A model can rewrite a word instead of copying it, and a rewritten word on a published page looks like the real one. bin/check-extraction compares every extracted title and description against the normalized source, field by field, through bin/canon.py. A curly apostrophe turned straight passes; a changed accent fails. That is the boundary the archive needs: it holds Catalan, Spanish, and French. bin/prompts/extract-talks.md is the prompt the model gets. It asks for a copy, never a correction, and says nothing about typography, because the normalizer owns that. tests/extraction/ records the measurement of 2026-10-11: 9 events from the export, 3 per format era, the talks each one announces read by hand, 7 real model runs, and the verdict the check has to give on each. One run has to fail, so the selftest proves the check still rejects a changed accent. The README of the directory says what each file is and why the recorded runs keep their old key names. pr.yml runs both selftests, and CONTRIBUTING.md lists five checks instead of three.
Five scripts under bin/, each reading an export that is not in this repository, so that every published fact of the archive can be traced to a source: - bin/import-meetups writes one page per monthly meetup out of the Meetup export: title, day, year, venue, registrations, source URL, and group. Each field is a copy of one field of the export, with canon() as the only transform. - bin/import-talks writes the talks block of each meetup page from a checked extraction, and resolves a speaker name to a person record by the same name, ignoring accents and case. A name that resolves to nothing stays in speaker_names, which no template reads. - bin/import-videos writes the video of a talk as a link of type video on its talk record, and the recording of a whole session as an entry of session_videos, from one normalized record per video of the channel. - bin/import-x-photos stages the photos PyBCN posted live from one event on X: the files under deterministic names, a photos block to paste into the front matter, and a report for the person who curates the set. A photo the association only shared is listed and never imported. - bin/list-channel reads the YouTube channel with yt-dlp and prints what the inventory does not hold, which is how a new video gets noticed. AGENTS.md says in what order the imports run, that they ran once, and what a re-run against content/ destroys. The bin/ row of its table names the new scripts. README.md indexes the two decisions of bin/import-x-photos: stage for review, never write content/, and never import a retweeted photo.
…o at 1000px bin/check-html-safety adds .gotmpl to the suffixes it scans. A content adapter, content/<section>/_content.gotmpl, is a template Hugo runs while it assembles the content, and it can write a page body through the content key of AddPage, which goes through goldmark like the body of a markdown file. No adapter is in the tree yet; the scan is ready for the first one. bin/fit-sources adds IMPORTED_PHOTO_CAP = 1000 for the photographs the archive imports. The gallery never shows one wider than 900px and the original is one click away in the post each one credits, so 1000 covers that at full detail and leaves the repository a fifth lighter. Its docstring also says that orpheus.png has been cut since the paragraph about it was written: 1120x269 and 16 KB, and a dry run over the set reports nothing.
LICENSING.md listed the Cookie Consent library by Osano among the vendored files and excepted cookieconsent.min.js from the MIT grant. Neither file is in the theme, so both sentences go. README.md said the repository is 139 MB with blobs up to 5.9 MB. Both figures move with every commit, so the paragraph now gives the command that measures them (git count-objects -vH) and no number. layouts/_default/_markup/render-link.html said that raw HTML no longer renders because goldmark unsafe is off. It is on, for the two pre-Hugo programmes pasted in as tables (AGENTS.md says so). The hook is still the reason a plain markdown link opens in a new tab; the comment now says why that is, without the false premise.
This was referenced Oct 15, 2026
Closed
ber2
approved these changes
Oct 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part 1 of 5 of the Archive, #213. The tools, with nothing of the archive itself: no page changes, no new content, and no template a reader reaches.
Everything here is a one-time importer or a check. The importers read an export of the Meetup, X, and YouTube accounts that is not in this repository, they were run once, and their docstrings say so:
bin/import-talksin particular is not safe to run twice, andAGENTS.mdrecords what a second run destroys. The archive they produced arrives in the four pull requests after this one.What it adds
bin/canon.py,bin/slug.py, andbin/titlematch.py: one definition of normalized text, of the URL form of a title, and of the comparison form used to find one title inside another. An imported string is then consistent by construction rather than by a model behaving well.bin/check-extraction: verifies an imported title and description against its source, character by character. Its test is seven recorded extraction runs, and one of them has to fail: a check that passes everything checks nothing.bin/import-meetups,bin/import-talks,bin/import-videos, andbin/import-x-photos, plusbin/list-channel, which asks our YouTube channel what it holds and names anything the site does not list. That last one is the only tool here meant to be run again.bin/check-html-safetylearns to read a content adapter, andbin/fit-sourcescaps an imported photograph at 1000px.bin/canon.py --selftestandbin/check-extraction --selftest.What it does not add
No check that depends on content this branch does not carry. A rule that rejects the old schedule model, or that checks a gallery, a venue, or the video inventory, lands with the content that makes it true: put it here and CI is red on arrival, which is measurable (the schedule rule alone gives 8 errors against
editiontoday).How to review
The diff is 29 files and no page of the site changes.
bin/check-content,bin/check-rendered, andbin/check-html-safetypass on this branch, and the build is the same 113 pages asedition.