Record the original for every rewritten URL attribute - #338
Draft
epheterson wants to merge 1 commit into
Draft
epheterson wants to merge 1 commit into
epheterson wants to merge 1 commit into
Conversation
Widens the script-src case to src, href and xlink:href on any tag. wombat's getAttribute override looks for the recorded original first and otherwise calls extractOriginalURL, which can only reverse rewrites wombat itself performed. A value rewritten here is unrecognisable to it, so it prepends the original scheme to what is already a relative path. That is openzim/warc2zim#413 as well as #473: same cause, two symptoms. Needs the matching wombat change, since retrieveWBOSRC is SCRIPT-only until then and nothing reads the other attributes back. Measured on en.wikipedia.org/wiki/Whale (1,766 links): 152 attributes added, 0.9% larger gzipped. Most links are fragments or already relative and record nothing. poster is deliberately excluded: wombat rewrites it, this rewriter does not, so it would record nothing.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #338 +/- ##
=========================================
Coverage 100.00% 100.00%
=========================================
Files 42 42
Lines 2694 2697 +3
Branches 398 397 -1
=========================================
+ Hits 2694 2697 +3 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Collaborator
|
Thank you very much! Just for clarification: this change concerns only ZIMs built with scraperlib rewriting, i.e. zimit/warc2zim and mindtouch ATM (the fact you tested on a WPEN page could lead someone to be concerned it might impact mwoffliner ZIMs, which is not the case - at this stage of mwoffliner development at least). Putting PR in draft until wombat PR settles, so we do not merge this by accident (even if it would be harmless, but not really useful either). |
benoit74
marked this pull request as draft
September 16, 2026 20:36
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #337, widening it from
<script src>tosrc,hrefandxlink:hrefon any tag, as you suggested.extractOriginalURLprepends the original scheme when it cannot recognise a value as one wombat rewrote, so an href rewritten here comes back ashttps://followed by a relative path.retrieveWBOSRCis SCRIPT-only and nothing reads the other attributes back.posteris excluded: wombat rewrites it, this rewriter does not, so it would record nothing.