LLM-as-Judge

Agent evaluation is the practice of proving that an AI agent completes its assigned task correctly and follows an acceptable path to get there. An agent you have not measured is unfinished: you cannot distinguish a regression from noise, you cannot adopt a better model without weeks of manual retesting, and you cannot answer the only question that matters after every change — did this help? Why an Unmeasured Agent Is an Unfinished Agent Traditional software ships when its tests pass. Agents rarely arrive with tests at all. They arrive with a demo, a trace viewer, and a feeling that the thing is working. ...

What Is OctopusGarden? — The Open-Source Dark Factory OctopusGarden is an open-source autonomous software development system that operates as a “dark factory” — it takes specifications written in markdown and scenarios defined in YAML, then builds the software without human intervention. Released by foundatron on GitHub under the MIT License, OctopusGarden represents one of the first fully open-source implementations of what Dan Shapiro calls Level 5 AI coding maturity: a system where humans define intent and review outcomes, but never touch the code during generation. The system is built in Go, requires Docker, and supports both Anthropic and OpenAI APIs. ...

Agent CI/CD in 2026 requires five evaluation gates that don’t exist in traditional pipelines: golden dataset offline eval, regression blocks, cost gates, shadow evaluation against production traces, and canary rollout with auto-rollback. If you’re shipping agent updates against only lint and unit tests, you’re shipping blind — 89% of production agent teams run observability but only 52% run evals, and that 37-point gap is where quality silently decays (LangChain State of Agent Engineering Survey, 2026). ...