Skip to content

reject jpeg scans using an undefined quantisation table - #1135

Open
metsw24-max wants to merge 1 commit into
ermig1979:masterfrom
metsw24-max:jpeg-undefined-dqt-table
Open

metsw24-max wants to merge 1 commit into
ermig1979:masterfrom
metsw24-max:jpeg-undefined-dqt-table

Conversation

@metsw24-max

Copy link
Copy Markdown
Contributor

Stale heap bytes decoded as pixels when a JPEG component has no DQT table

JpegContext::dequant is never initialised (the context comes from new JpegContext and the constructor only sets stream and img_n), and the Avx2 jpeg__jpeg is malloc'ed. Neither decoder checks that the table a component selects in SOF (tq) was actually loaded by a DQT marker, so a file that omits the DQT, or leaves out one of its tables, is dequantised with whatever the allocator left in those 512 bytes. JpegDecodeBlock multiplies every coefficient by it (JpegFinish does the same for progressive files) and the IDCT output comes back from SimdImageLoadFromMemory as pixels.

A 64x64 JPEG written by SimdImageSaveToMemory with its DQT segment removed still decodes on both decoders. With the context memory pre-filled with 0x00 every Gray8 pixel is 128; with 0x01 or 0xA5 the output follows the fill (Base pixel sums 532290 and 519411, Avx2 516617 and 516829). Dropping only the chroma table from a colour JPEG does the same to the chroma planes.

Fix: record which tables a DQT defined (cleared with the Huffman flags at header start) and reject a scan whose components point at an undefined one, which is where libjpeg raises JERR_NO_QUANT_TABLE. The Avx2 decoder gets the same change, as it keeps its own marker and scan-header parsers.

Notes: 187 valid JPEGs (the three bundled samples, 112 baseline and progressive files written by macOS ImageIO, 72 from SimdImageSaveToMemory) decode bit-identically before and after in all five output formats, through Simd/Base on arm64 and Base/Sse41/Avx2 on x86_64 under Rosetta (Avx2::ImageLoadFromMemory called directly). Test -fi=ImageLoad -fi=ImageSave passes. In the full -m=a run, RecursiveBilateralFilterAutoTest fails with the same 241 vs 240 mismatch on unmodified master on this machine (Apple M5, Apple clang), so I excluded it; everything else passed up to 91%, where I stopped the Gemm32f, DescrIntCosineDistancesMxNa and SynetDeconvolution32f groups after about an hour in reference code. They do not touch the loader, but a CI run would still be worth having. The check runs once per scan, so decode speed should be unaffected.

Neither JPEG decoder initialises its dequant tables, and nothing checks that the table a component selects in SOF was loaded by a DQT marker, so a file without that DQT was dequantised with stale heap bytes that ended up in the output pixels. Track which tables were defined and reject such a scan at scan start, as libjpeg does.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant