Real SM Utilization for NVIDIA: When nvidia-smi Lies
nvidia-smi’s GPU-Util is a duty cycle, not a measure of work. It reports the percentage of the last sample window during which at least one kernel was running — nothing about how many of the GPU’s streaming multiprocessors were busy, how full they were, or whether the Tensor Cores did anything. Real NVIDIA SM utilization comes from DCGM’s profiling fields: DCGM_FI_PROF_SM_ACTIVE (breadth), DCGM_FI_PROF_SM_OCCUPANCY (depth), DCGM_FI_PROF_PIPE_TENSOR_ACTIVE (the work you pay for) and DCGM_FI_PROF_DRAM_ACTIVE (the bottleneck). Two of those four are commented out of dcgm-exporter’s default counter file. ...