Repository navigation
fix: integrate GPU power when energy counter is absent - #1319
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #1319 +/- ##
==========================================
+ Coverage 92.15% 92.36% +0.20%
==========================================
Files 49 49
Lines 5227 5341 +114
==========================================
+ Hits 4817 4933 +116
+ Misses 410 408 -2 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
4e36cc8 to
1889fda
Compare
Verdict: ✅ Approve with nitsThe bug is real. On master,
54 GPU tests pass on this PR merged with master. Nits / follow-ups:
|
|
Made the changes in 19613ce: merged master, set static details before the first energy read (AMD dual-GCD flag), and updated the description to list all 4 tests. |
GPUs without nvmlDeviceGetTotalEnergyConsumption now integrate instantaneous power instead of reporting zero, re-baselining cleanly when the counter returns. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
AMD dual-GCD devices set _known_zero_energy_counter there, so the secondary GCD no longer takes the power fallback on its first interval. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- Sample GPU power every second in the tracker's monitoring loop while the energy counter is unavailable, and average the samples over the interval instead of using a single reading taken at its end. - Keep the last good counter reading during a counter outage and, when it comes back, report the counter delta minus what the fallback already reported, so the recovered interval uses the counter again. - Treat a counter going backwards (driver reload, reset) as a fallback interval and re-baseline, instead of reporting negative energy. - Use class-level defaults for the fallback flags so devices built without the dataclass __init__ do not raise AttributeError. - Warn once per device when power cannot be read or reads 0 W without a counter; warn once per outage when a working counter fails, and log its recovery. - Lower the per-sample AMD energy counter warnings to debug, like NVIDIA. - Document the fallback in docs/explanation/power-estimation.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017Bj3CXQU2KcRKCxrb17j9e
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Instance state was declared as class attributes so that instances built with __new__ in tests still worked, and _power_samples was never declared, which required getattr guards on the hot path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
start() re-read the energy counter without updating _use_power_fallback. A transient read failure at start() left the flag False, so no power was sampled during the first interval, and delta() then logged the permanent 'no energy counter' warning for a device that has one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Clamping the recovered delta to 0 dropped any energy the power fallback over-reported, so it stayed in the totals forever. The excess is now deducted from the following counter deltas. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On counter recovery, the counter delta covers every fallback interval of the outage, but the power was computed over the last interval only. After a long outage this reported a power several times the real draw. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sample_power() runs in the 1s monitoring thread while delta() swaps the sample list in the measurement thread. Without a lock a sample could be appended to the discarded list and lost, or appended after the counter came back and leak into a later fallback interval. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The second GCD of MI250/MI300 devices reports 0 W by design, its power is counted by the first GCD. In fallback mode it logged a misleading under-reporting warning. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
self.devices is always an AllGPUDevices, which defines sample_power. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The 1s monitoring loop read the power of every GPU, ignoring gpu_ids, so a node tracking one GPU out of 8 made 8 power calls per second. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Counter read errors were moved to debug level, so at the default level the fallback warning did not say why the counter could not be read (permissions, driver error...). The backend error is now appended to it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The GPU power fallback relies on the 1s monitor to sample power, but stop_task() stopped it for good after start(), and start() only started it when output handlers existed. Later intervals fell back to a single power reading. The monitor is now always started by start(), and resumed with the measurement scheduler after a task. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
FakeGPUEnv deleted codecarbon.external.hardware on teardown and left the GPU modules imported against the fake pynvml. Test files running after test_gpu_nvidia.py then re-imported hardware and got other GPU/CPU classes than the ones codecarbon.emissions_tracker holds, so isinstance checks and mock.patch targets missed: 5 tests in test_emissions_tracker.py failed when run right after it. The modules are now snapshotted and restored. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
33fce8d to
34e9ae2
Compare
Description
GPUDevice.deltanow branches on the backend returningNonefor the cumulative energy counter. When it does, it computesPower.from_watts(self._get_power_usage())andEnergy.from_power_and_time(...)instead of comparing two identical counter readings, and logs a single warning per device so users know which method produced their numbers. The per-samplelogger.warninginNvidiaGPUDevice._get_total_energy_consumptionis demoted todebug, since the once-per-device warning now carries the signal.Related Issue
Fixes #1316
Motivation and Context
On GPUs where NVML does not implement
nvmlDeviceGetTotalEnergyConsumption(pre-Volta cards, many virtualised GPUs) the previous code returnedlast_energyunchanged, yieldingenergy_delta = 0andpower = 0for the entire run, so a GPU-bound job silently reported only CPU + RAM. The fallback lives in the sharedGPUDevice, so the AMD backend (which also returnsNonewhen no energy accumulator is present) is fixed by the same change. Power integration is less accurate than the counter, but it is the same approximation the CPU path already uses and is dramatically better than zero.How Has This Been Tested?
tests/test_gpu_nvidia.py::TestGpu::test_gpu_without_energy_counter_falls_back_to_powerpatches the NVML energy query to raise and asserts 26 W / 29 W and 0.026 / 0.029 kWh over an hour. It fails on master (assert 0.0 == 26) and passes here. Three more tests intests/test_gpu_nvidia.pycover the rest:test_gpu_energy_counter_recovery_does_not_double_count,test_gpu_without_power_usage_reports_zero_instead_of_raisingandtest_start_rebaselines_the_energy_counter.tests/test_gpu.py,tests/test_gpu_nvidia.pyandtests/test_gpu_amd.pyall pass (54 tests).Screenshots (if appropriate):
N/A
Types of changes
AI Usage Disclosure
Checklist: