[putdanil]

Speeds up GPU inference by converting uint8 → float32 on the device instead of the CPU; outputs are unchanged.

PTDetector._process_batch_group() converts each batch to float32 and divides by 255 on the CPU, then copies it to the device. On CUDA devices, this PR copies the uint8 tensor first and converts it on the GPU: the copy is 4x smaller, and the per-pixel conversion no longer runs on the CPU inside the inference loop.

The division uses a 0-dim tensor on the device rather than the Python float 255.0. CUDA computes x / scalar as x * (1 / scalar), which differs from the CPU result in the last bit for 126 of the 256 pixel values; dividing by a device tensor is a true division and gives exactly the CPU's float32 values (checked for all 256 values). Other devices (CPU, MPS, DirectML) keep the existing code path, since I couldn't test them. If you can test on MPS or DirectML, a 256-entry lookup table computed on the CPU (lut[x.long()]) would be exact on any device.

Measurements

MDv1000-redwood, run_detector_batch CLI, 500 images from LILA (mixed resolutions), an NVIDIA GPU + 8 vCPU (Xeon Gold 6248R VM). Images/s excluding start-up (from runs on 100 and 500 images):

configuration	main	this PR
default	6.9	7.7
--batch_size 8	10.3	10.5
--use_image_queue --preprocess_on_image_queue, with #256	12.6	16.5
same with --batch_size 8, with #256	24.3	34.0

Single runs; run-to-run spread on this machine is a few percent. The gain is largest once the queue no longer limits throughput (#256).

Equivalence

The images list of the output file is identical to main's for default and --batch_size 8, and for augmented inference (the md_tests augmentation step, 47 images; dividing by the Python float instead changed 5 of them in the last digit). With --use_image_queue --preprocess_on_image_queue --batch_size 8, main itself is not deterministic run to run (batches follow the order in which images arrive from the loaders; two runs of main differ on 22 of 500 images, confidence in the last digit, no box changes), and this PR differs from main by the same amount.

Tests

Run on main and with all three of my PRs applied (#255, #256, #257), on the same machine:

• pytest -vv: 24 passed, 4 skipped, on both.
• python -m megadetector.utils.md_tests --max_coord_error 0.01 --max_conf_error 0.01 --skip_download_tests --warning_mode (GPU, CPU, and video tests): all 35 steps run on both, and the comparison lines against the expected results are the same. 40 of the 41 result files are identical to main's; the remaining one (worker-side preprocessing with batched inference, the non-deterministic case above) differs on 1 of 47 images by 0.001 in confidence.

--warning_mode is there because, independently of these PRs, md_tests fails on main at "Running MD on a folder of videos": DSCF0028.MP4 is reported with a confidence error of 0.667, although every box above 0.5 matches the expected results at IoU ≥ 0.998 with confidence within 0.021 (and the assertion message names two boxes with confidence 0.005). Presumably the frames decode slightly differently here: opencv-python 5.0.0 (FFmpeg avcodec 62.28.101), torch 2.14.1+cu130, NVIDIA driver 580.105.08, Ubuntu 22.04, Python 3.11.14.

---

This came out of benchmarking run_detector_batch end to end (related to the TODO item "Performance evaluation for batching, preprocessing"); happy to share the full numbers. Prepared with help from Claude Code; I reviewed the diff and the measurements.

[agentmorris]

Thanks! Merging via PR 259 (minor comment edits only, and merging main).