Repository navigation
Follow m3u playlists under every content type servers send, HLS included - #2231
Merged
Merged
Conversation
An m3u playlist was parsed for links only when served as audio/x-mpegurl. Python's http.server sends audio/mpegurl, and HLS playlists come as application/vnd.apple.mpegurl or application/x-mpegurl, so the files they list were never fetched. Playlist mode also scanned "#" lines for links, so every extended playlist asked the server for "EXTM3U" and had its "#EXTM3U" header rewritten to "#EXTM3U.html". Those lines are now skipped. Closes #2228 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
In playlist mode every byte could start a link, and the link code skips whitespace including newlines. A link starting on a line end or a blank line therefore ran into the next line, so a "#" line after another "#" line or after a blank line was fetched with its "#" dropped, and a trailing "#" line gained "#" and a NUL in the mirror. A link now starts only at the first non-blank byte of a line, a "#" line is skipped to its end, and a link also ends at a CR so CR-only playlists work. Test 2271 now serves each playlist type with "#" lines, blank and whitespace-only lines, a "#" inside a media line, and LF, CRLF and CR-only line ends, and checks the mirror byte for byte. Based on the fix in 45291948 (playlist-comment-lines). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
A link starts only at a line's first non-blank byte, so excluding '#' there is enough: the skip loop to the end of the line is not needed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
…type A .m3u8 served as application/octet-stream is now parsed, because its extension maps to application/vnd.apple.mpegurl. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
xroche
changed the title
A link that runs to the end of the buffer stops on its NUL. The main loop then stepped past it and flushed that NUL into the mirror, so the scan pointer is now clamped to the buffer end. A UTF-8 BOM at the start of a playlist was read as the first link, so the first line now starts after it. The CR end of a playlist link is gated on LNK mode by name. Test 2271 now covers these cases: - a last line with no line end, after CR and after LF - a BOM - a line with two names, which gives one link - an octet stream with no playlist extension, which is not scanned Signed-off-by: Xavier Roche <roche@httrack.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
The scan pointer is clamped to the buffer end so that a link at EOF does not write the buffer's NUL. A base href or applet codebase is skipped rather than written, and its write mark then sat one byte past the NUL, after the clamped pointer. The flush then asked for a negative length and aborted. The write mark is now clamped with the scan pointer. Fuzz seeds and test 443 cover a page that ends inside a base href or an applet codebase value. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
xroche
enabled auto-merge (squash)
November 14, 2026 12:49
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
xroche
added a commit
that referenced
this pull request
Nov 14, 2026
…#2234) An abort used to download or probe every remaining link on the page before it stopped, because the waits for a link's type ignored it. The engine now stops at once and leaves those links online, and `--continue` fetches them. A soft stop (the first ^C, or `hts_request_stop(opt, 0)`) is unchanged. Only hard stops (the abort lock, SIGTERM, `hts_request_stop(opt, 1)`) leave the remaining links for `--continue`. Test 436 failed on a macOS runner on #2231, because it passed only when the last file happened to be cut. It now checks that at most the first two files exist after the abort, and that `--continue` completes the mirror. Test 262 covers SIGTERM during the `-%N0` type probe. --------- Signed-off-by: Xavier Roche <roche@httrack.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
HTTrack now follows the links in an m3u playlist served as
audio/mpegurl,application/x-mpegurlorapplication/vnd.apple.mpegurl. The.m3u8extension now maps toapplication/vnd.apple.mpegurl, so a.m3u8file sent asapplication/octet-streamis followed too. A playlist link starts only at the first non-blank byte of a line and never on a#line. So a line gives at most one link, even when it holds two names. CR-only line ends and a UTF-8 BOM work. A page of any type that ends inside a link no longer gets a NUL added to its mirror.Closes #2228
Closes #2230