AA AgentPerf Local: What Replaying Real Agent Trajectories Measures (and What It Cannot)
AA-AgentPerf-Local is an open-source tool from Artificial Analysis that replays recorded agent trajectories against a local OpenAI-compatible server and reports throughput and latency. The short answer to what it measures: serving speed on agent-shaped traffic - requests that carry the whole conversation so far, with long prompts and heavy key-value cache reuse - and deliberately nothing about output quality. The short answer to what it cannot do: it cannot give you a portable number. A result from it is only comparable if the context size was full, the output policy was exact, and the tool mode is stated. Miss any of those and you have an indicative run, not a benchmark. ...