Repository navigation
Make an abort stop at once instead of downloading the remaining links - #2234
Merged
Merged
Conversation
An abort read from hts-abort.lock (or a SIGTERM) sets exit_xh, but the parser's per-link type wait and the socket wait it starts with never looked at it. With -c1 each link then waited for the previous transfer, so the abort landed only after the whole page was downloaded. Test 436 passed only because the last file was usually cut a few KB in; on a macOS runner it had no body yet, so no ref was kept (seen on #2231). Both waits now give up on an exit, the link is left online as with -E, and test 436 checks that the last file is never fetched. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
The -%N0 type probe in url_savename() checked only _hts_cancel and the -E budget, so after SIGTERM the parse still probed the remaining links one by one and then fetched them (13s on the new /slowhead/ fixture, 0.1s now). Test 262 covers it. Test 436 now asserts that only p0 and p1 can exist after the abort, at most one of them complete, with no temporary file left, and that a --continue finishes all eight files with local links. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
xroche
changed the title
xroche
enabled auto-merge (squash)
November 14, 2026 14:37
xroche
added a commit
that referenced
this pull request
Nov 14, 2026
An abort lock written during the `-%N0` link-type probe now stops the crawl at once. Before, the crawl first probed every remaining link, which took 14s in test 436. This is the lock-file half of #2234, which fixed the same wait for SIGTERM. I also tried a poll in the slot wait (`hts_wait_available_socket`) and dropped it, because no crawl I could build needed it. --------- Signed-off-by: Xavier Roche <roche@httrack.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
An abort used to download or probe every remaining link on the page before it stopped, because the waits for a link's type ignored it. The engine now stops at once and leaves those links online, and
--continuefetches them. A soft stop (the first ^C, orhts_request_stop(opt, 0)) is unchanged. Only hard stops (the abort lock, SIGTERM,hts_request_stop(opt, 1)) leave the remaining links for--continue.Test 436 failed on a macOS runner on #2231, because it passed only when the last file happened to be cut. It now checks that at most the first two files exist after the abort, and that
--continuecompletes the mirror. Test 262 covers SIGTERM during the-%N0type probe.