Do Local Coding Models Actually Finish Agent Tasks? A Reproducible Task-Success Protocol
Choosing between a local open-weight coding model and a managed agent is usually argued with the wrong number. Throughput, price per token and leaderboard position are all published and all easy to compare; none of them answers the question that decides the purchase: on this kind of task, on hardware like mine, how often does the model actually finish before it stalls, loops or asks for help? This article ships a small protocol that measures exactly that, and reports no model results of its own. It publishes a frozen four-task sandbox, a harness with a documented adapter contract, a scoring sheet and a fixture self-check whose raw output is recorded and hashed. The model-by-task matrix in the middle of this page is filled with not run, because no model was executed while writing it. That is the honest state of the evidence: the instrument is built and verified, the vendor numbers it compares against are third-party, and the first real row is generated by whoever points it at a served model. ...