Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 30 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

33 changes: 33 additions & 0 deletions crates/clazel-webp/Cargo.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
[package]
name = "clazel-webp"
version = "0.0.0"
edition.workspace = true
license.workspace = true
publish.workspace = true
description = "A WebP decoder (Phase 6 item 6.8): the RIFF container, VP8X, ANIM and ANMF with every frame composited, and VP8L lossless decoding (transforms, colour cache, meta prefix codes, backward references); lossy VP8 is item 6.9 and is refused. A resumable step function over caller-owned buffers, implementing clazel-image-shape's Decoder."

[dependencies]
# The decoder interface (item 6.3), our own crate with no dependencies, so no
# third-party code is shipped (CLAUDE.md rule 8).
clazel-image-shape = { path = "../clazel-image-shape" }

# The reference decoder of the differential test (tests/conformance_webp.rs),
# never compiled into the browser: image-rs's `image-webp`, written by others,
# driven for the pixels and frames of each file (docs/DEPENDENCIES.md, class
# `test`).
[dev-dependencies]
image-webp = "=0.2.4"

# Verification tier (docs/VERIFICATION.md). `tier` is what is proven about the
# shipped code today; `target-tier` is the plan's reading of item 6.8: bytes
# of a page-chosen file drive every index and counter, so Tier 1 (Kani
# harnesses in src/proofs.rs: prefix code construction over any code lengths,
# each transform over any row up to a stated width, and the step function's
# parts). The tier stays 0 until CI's full verification has run those
# harnesses green.
[package.metadata.clazel]
tier = 0
target-tier = 1

[lints]
workspace = true
159 changes: 159 additions & 0 deletions crates/clazel-webp/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,159 @@
# clazel-webp

A WebP decoder for [Clazel](../../README.md), a web browser whose parsers of
untrusted input carry machine-checked proofs. Phase 6 item 6.8
(`docs/phases/phase-6.md`): the RIFF container, animation, and **lossless**
(VP8L) decoding. Lossy VP8 is item 6.9; here a `VP8 ` chunk is recognised and
refused with `Error::Unsupported`. It runs in the renderer's sandbox, on bytes
a page chose, and implements the one interface of every Phase 6 decoder,
`clazel_image_shape::Decoder` (item 6.3): a resumable step function over
caller-owned buffers, nothing allocated while decoding.

Its only dependency is `clazel-image-shape`. `image-webp` is a dev-dependency,
used only by the differential test.

## What it does

RFC 9649 (the WebP container, and the lossless bitstream): `RIFF`/`WEBP`, the
simple lossless format, the extended format (`VP8X`; `ICCP`, `ALPH`, `EXIF`,
`XMP ` and unknown chunks skipped), `ANIM` and `ANMF`. **Every frame** of an
animation is decoded and composited onto the canvas with the previous
frame's disposal and the frame's blending method; after each
`Status::FrameDone` the pixel buffer holds the canvas as displayed with that
frame and `Webp::frame()` has the frame's rectangle, duration, blending and
disposal. VP8L: the four transforms (predictor with its fourteen modes,
colour, subtract green, colour indexing with pixel bundling), the colour
cache (1 to 11 bits), meta prefix codes, simple and normal code length codes,
LZ77 backward references with the distance map.

| File | Holds |
|------|-------|
| `src/lib.rs` | the pinned standard, the section numbers, the listed choices |
| `src/decoder.rs` | the container state machine, `Webp`, `Decoder`, `Webp::state_inv`, compositing |
| `src/vp8l.rs` | the lossless bitstream as a state machine over atomic actions |
| `src/huffman.rs` | canonical prefix codes: construction, decoding without tables |
| `src/transform.rs` | the inverse transforms, one row at a time, in place |
| `src/distance.rs` | the LZ77 prefix coding and the distance map (Figure 20) |
| `src/compose.rs` | alpha blending |
| `src/bits.rs`, `src/work.rs` | the bit reader, the work buffer's layout |
| `src/proofs.rs` | the Kani harnesses |

The work buffer is `work_len(config)`: about 11.6 kB (colour cache, code
lengths, colour table), 12 bytes per block of 4 by 4 pixels (the predictor,
colour transform and entropy images), 6.4 kB per prefix code group held (at
most 1,024, and at most one per block of 4 by 4), and for an animation a
frame buffer as large as the canvas. A still image is decoded straight into
the pixel buffer.

## Listed choices (`src/lib.rs` has them with the section numbers)

Animation canvas transparent black and the `ANIM` background ignored,
disposal to transparent; alpha blending on stored values with libwebp's
integer arithmetic (a destination of alpha 0 takes the source exactly);
`alpha_is_used` ignored; `ANMF`/`ANIM` skipped in a still image, image data
outside a frame in an animation an error; `ALPH` skipped; the RIFF size ends
the stream; prefix codes empty, over-subscribed or incomplete are errors and a
single symbol takes zero bits; predictor modes 14 and 15 predict like mode 0;
at most 1,024 prefix code groups (more is `Unsupported`); a backward
reference before the first pixel or past the last is an error.

## Kani harnesses (`src/proofs.rs`)

Sixty-nine. The step harnesses are one per phase, role or transform: with
the phase symbolic, CBMC ran out of memory (over 4 GB locally; on CI the
runner was lost after two hours). What keeps each in memory is listed at the
top of `src/proofs.rs` (MiniSat for the pixel, finish and container ones,
smaller unwinding, a 4-colour table where it is indexed symbolically).
**Not yet run green on CI**: locally, under Kani 0.68.0 with a 4 GB cap, each
was run on its own (results in the pull request); CI's approved proof jobs
are the result that counts.

| Harness | Statement (every one: no panic, no overflow, no out-of-bounds index) | Bound |
|---------|-----------|-------|
| `huffman_build_any` | prefix code construction: an error if a length is over 15; otherwise complete (Kraft sum exactly 1) or a single symbol, each used symbol once in the table | **any code lengths**, 6 symbols |
| `huffman_decode_any` | decoding: a symbol of the alphabet with the code's length, never more bits than available, `More` only under 15 bits | any built code, any 32 bits, any count available |
| `distance_value_any` | the LZ77 prefix coding: 1 to 1,048,576 | every prefix code 0 to 39, any extra bits |
| `distance_map_any` | the distance of a code is at least 1, `code - 120` above 120, within 7 rows and 8 columns below | every code, every width up to 2^14 |
| `predict_any` | each predictor mode's value (L, T, TR, TL, black, Average2 of L and T, Select is L or T) | every mode, any four pixels |
| `predictor_row_any` | the inverse predictor of a row: the other row untouched, the top-left pixel is black plus residual | any row of 3 pixels in 2 rows, any block bits 2 to 9, any modes, any pixels |
| `color_row_any` | the inverse colour transform: green and alpha unchanged | any row of 3, any transform data |
| `green_row_any` | the inverse subtract green: exactly red and blue gain green | any row of 3 |
| `palette_row_any` | colour indexing with bundling: each pixel is the table entry its packed bits name, or transparent black past the table | every bundling width, rows of 1 to 8 pixels, up to 4 colours |
| `palette_in_place_any` | the same, two rows expanded in place, last row first, equal to the expansion from a copy | every bundling width, 4 pixels in 2 rows |
| `blend_px_any` | alpha blending: alpha 255 gives the source, 0 the destination, otherwise alpha does not go down | any two pixels |
| `act_*` (40, `act_harness!`) | one atomic action of the lossless decoder, in one phase each (header, transforms, cache, meta prefix, every step of reading a prefix code, and pixels: literal symbols, lengths, distances, cache lookups, a backward reference of at most 4 pixels), keeps `Vp8l::inv` on `Ok`; the two distance phases have one harness per role | any state with the invariant, any bits, any contents of every scratch region; images of at most 4 pixels; `act_build_code`: an alphabet of at most 8 symbols; palette role: a table of 4 colours |
| `finish_*` (9) | finishing an image keeps `Vp8l::inv`: the colour table, entropy image and two transform images, and the main image with no transform or with one of each kind (inverting several is a loop whose body depends only on that transform and the pixels) | the same, up to 4 colours, 2 rows; `finish_main_predictor` exactly 2 by 2 pixels (`predictor_row_any` covers rows of 3); `finish_main_index`: a table of 4 colours |
| `container_*` (9, `container_harness!`) | one byte of the container, in each phase but the lossless payload, keeps `Webp::state_inv` on `Ok` | any state with the invariant, a 2 by 2 canvas |

### What Kani covers, and what it does not

- **The step function itself is not machine-checked, its parts are.** Item
6.3's template `decoder_step_harness!` was not instantiated: GIF's
experience (item 6.7) is that CBMC does not finish on a whole state machine
of this size, and this one is larger (a work buffer of 15 kB even at the
harness's sizes). `step` is `Parts::new` (a unit test and the lengths in
`inv`), then a loop of `Webp::byte`, which is `dispatch` (the container
harness) and, in a lossless payload, `Vp8l::push` and `Vp8l::advance`, a
loop of `Vp8l::act` (one harness per phase) and `finish_image` (one per
role). What the template would add and is argued here, not
proven: that `inv` is kept over several bytes (induction from one), that
`consumed <= chunk.len()` and that `NeedMore` consumes the chunk (a loop's
index), and that nothing outside the buffers is written (safe Rust
bounds-checks every index, and no index uses unchecked arithmetic: the
workspace lints deny it).
- **Small bounds.** Under Kani the colour cache is 8 entries (`work.rs`), so a
green alphabet is 288 symbols; images are at most 4 pixels; a backward
reference at most 4 pixels. The code is the same code with other
constants; the real sizes are run by the unit tests, the differential test
and Miri.
- **Separate arrays**: the harnesses give the scratch regions as separate
arrays, not one work buffer cut by `Parts::new`.
- **Not stated**: that the pixels are right (the differential test's and the
hand-made streams'), that the work is bounded in time (an image of 2^26
pixels coded with zero-bit prefix codes decodes from a handful of bytes:
the limits of item 6.0(j) bound it), and that every state `inv` admits is
reachable (it over-approximates, so the harnesses are sound).
- The tier stays 0 until CI's full verification has run these green
(CLAUDE.md rule 7).

## Tests

- **Unit tests** (22): the bit reader, the layout, prefix codes (the RFC
example, one symbol, bad codes), the distance map and Table 4, each
transform and predictor border, blending.
- **`tests/conformance_webp.rs`** (15 tests): Wuffs' WebP test data
(`tests/data/wuffs-webp`, commit `b36a81d`, MIT or Apache-2.0 at our
choice; libwebp's own vectors have no licence and are not vendored)
against `image-webp` 0.2.4: **the 11 lossless files agree, exactly**
(the largest 1,165 by 859); the animation agrees on its first frame and
within 1 per channel on the alpha-blended ones (`image-webp` rounds blending
more precisely than libwebp's formula, which is followed: listed); the 2
lossy files are refused as unsupported, each with its reason. Every file at
chunk sizes 1 to 4096 (with and without empty chunks) equals the
whole-file decode, with the invariant checked at every step and canary
bytes past the buffers untouched. 1,440 deterministic byte mutants
never panic (968 end in an error, the rest decode something).
- **`tests/handmade/`**: a VP8L encoder written for the tests (from the RFC,
sharing no code with the decoder) makes **531 images** that must decode to
exactly the source, byte by byte, whole, and as `image-webp` reads them:
ten orders of transforms at six sizes and six seeds (360), every predictor
mode (6), pixel bundling at every palette size from 1 to 256 and nine
widths (81), caches of 0 to 11 bits with and without meta prefix codes (72),
backward references of every distance code and simple and normal prefix
codes (12). Container cases: a still `VP8X` with chunks to skip and
trailing data; an animation of five frames with blending, no blending and
both disposals, checked against a compositor written in the test; eight
malformed files and four malformed streams, each refused with the right
reason.
- **Miri**: every test runs a stride (`cfg!(miri)`): the 156-byte file, chunk
size 1, 2 mutants, one hand-made image of at most 8 by 4 pixels. Miri runs
25,000 to 30,000 times slower than native; measured with `--cfg miri`
natively, the 156-byte file decodes in 0.85 ms.

## Not done

Lossy VP8 and `ALPH` (item 6.9). The loop count and frame timing policy
(6.19). Prefix code group counts above 1,024 (or above the image's blocks).
libwebp's own test vectors (no licence) and `dwebp`'s output: `image-webp`
is the reference, and libwebp's blending is reproduced from its source's
formula, not compared with it. Kani has not been run on this crate.
125 changes: 125 additions & 0 deletions crates/clazel-webp/src/bits.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,125 @@
//! The bit reader: bits of the stream, least significant first (RFC 9649
//! section 3.3's `ReadBits`), pushed one byte at a time.
//!
//! RFC 9649 section 3.3 ("Nomenclature") and 3.7.2.3: bits are read from the
//! least significant bit of each byte first.

/// The unread bits of the stream. A byte is pushed only when 8 more fit in
/// the 64-bit accumulator; the decoder drains it to under 32 bits after
/// every push.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub(crate) struct Bits {
/// The unread bits, the next one in bit 0. Bits at and above `n` are 0.
acc: u64,
/// How many bits of `acc` are unread (0 to 64).
n: u8,
/// The last byte of the chunk was pushed: no more bits will come.
pub(crate) end: bool,
}

impl Bits {
/// An empty reader.
pub(crate) const fn new() -> Bits {
Bits {
acc: 0,
n: 0,
end: false,
}
}

/// The invariant: the accumulator holds no bit at or past `n`.
pub(crate) fn inv(&self) -> bool {
self.n <= 64
&& self
.acc
.checked_shr(u32::from(self.n))
.is_none_or(|rest| rest == 0)
}

/// How many bits are unread.
pub(crate) fn len(&self) -> u8 {
self.n
}

/// Appends the next byte of the stream. (Does nothing if the
/// accumulator is too full; the decoder drains it after every push, so
/// that does not happen.)
pub(crate) fn push(&mut self, b: u8) {
if self.n <= 56 {
let shifted = u64::from(b).checked_shl(u32::from(self.n)).unwrap_or(0);
self.acc |= shifted;
self.n = self.n.saturating_add(8);
}
}

/// The next 32 bits (bits not yet pushed read as 0), the next one in
/// bit 0.
pub(crate) fn peek32(&self) -> u32 {
u32::try_from(self.acc & 0xFFFF_FFFF).unwrap_or(0)
}

/// Drops `k` bits (at most the unread ones).
pub(crate) fn skip(&mut self, k: u8) {
let k = k.min(self.n);
self.acc = self.acc.checked_shr(u32::from(k)).unwrap_or(0);
self.n = self.n.saturating_sub(k);
}

/// Reads `k` bits (0 to 32) if that many are unread.
pub(crate) fn take(&mut self, k: u8) -> Option<u32> {
if k > 32 || self.n < k {
return None;
}
let mask = if k == 0 {
0
} else {
u32::MAX
.checked_shr(u32::from(32u8.saturating_sub(k)))
.unwrap_or(0)
};
let v = self.peek32() & mask;
self.skip(k);
Some(v)
}
}

#[cfg(kani)]
impl Bits {
/// Any reader, the invariant not assumed.
pub(crate) fn any() -> Bits {
Bits {
acc: kani::any(),
n: kani::any(),
end: kani::any(),
}
}
}

#[cfg(test)]
mod tests {
use super::*;

#[test]
fn reads_lsb_first() {
let mut b = Bits::new();
b.push(0b1010_0110);
b.push(0xFF);
assert_eq!(b.take(3), Some(0b110));
assert_eq!(b.take(5), Some(0b10100));
assert_eq!(b.take(8), Some(0xFF));
assert_eq!(b.take(1), None);
assert!(b.inv());
}

#[test]
fn zero_and_wide_reads() {
let mut b = Bits::new();
for _ in 0..4 {
b.push(0xAB);
}
assert_eq!(b.take(0), Some(0));
assert_eq!(b.take(32), Some(0xABAB_ABAB));
assert_eq!(b.len(), 0);
assert_eq!(b.take(33), None);
}
}
Loading
Loading