Agent Testing

ASSERT (Adaptive Spec-driven Scoring for Evaluation and Regression Testing) is an open-source, requirement-driven evaluation harness for AI agents and LLM applications, announced at Microsoft Build 2026. Built by Microsoft’s Responsible AI organization and released under the MIT license, ASSERT transforms natural-language agent policies into executable test suites through a four-stage pipeline — systematize, test set, inference, and judge — making it the first framework to bridge the gap between product requirements and automated test coverage for AI agents. ...

You demo an AI agent to your team. Fifty runs, zero failures. Everyone’s impressed. You deploy to production. Within a week, it’s hallucinating tool calls, getting stuck in loops, and your Slack is full of “the agent did something weird” messages. I’ve been there. Multiple times. And I’ve spent the last year digging into why this happens and what actually works to fix it. The short answer: your agent isn’t broken — your testing methodology is. Single-digit demos and pass/fail judgments hide a massive variance problem that only emerges under statistical scrutiny. Gartner predicts over 40% of AI agent projects will fail by 2027, and in January 2026, a prompt injection in a customer support agent processed a $47,000 fraudulent refund. These aren’t edge cases — they’re systematic failures that most teams aren’t testing for. ...

You demo an AI agent to your team. Fifty runs, zero failures. Everyone’s impressed. You deploy to production. Within a week, it’s hallucinating tool calls, getting stuck in loops, and your Slack is full of “the agent did something weird” messages. I’ve been there. Multiple times. And I’ve spent the last year digging into why this happens and what actually works to fix it. The short answer: your agent isn’t broken — your testing methodology is. Single-digit demos and pass/fail judgments hide a massive variance problem that only emerges under statistical scrutiny. Gartner predicts over 40% of AI agent projects will fail by 2027, and in January 2026, a prompt injection in a customer support agent processed a $47,000 fraudulent refund. These aren’t edge cases — they’re systematic failures that most teams aren’t testing for. ...

AWS Agent-EvalKit is an open-source toolkit (Apache 2.0, released June 11, 2026) that runs AI agent evaluation directly inside your coding assistant via slash commands. Instead of treating agent evaluation as a post-deployment activity, it brings a six-phase workflow — Plan, Data, Trace, Run Agent, Eval, Report — into Claude Code, Kiro CLI, or Kilo Code, combining code-based evaluators with LLM-as-judge scoring through Amazon Bedrock. I’ve been running evaluations against AI agents for the last two years, and the pattern I kept seeing was: teams either buy a managed eval platform or cobble together Python scripts and a prompt template. Agent-EvalKit splits the difference — it’s a CLI that reads your agent source code, generates test cases, instruments tracing, runs the trials, and recommends fixes with file-level accuracy. In this tutorial, I’ll walk through installing it, running your first evaluation, and the real-world case study where it caught a hallucination problem that output-level testing missed entirely. ...

LangWatch is an open-source monitoring, evaluation, and optimization platform for LLM applications and AI agents. It provides tracing, real-time evaluation, agent simulation, and prompt management in a single unified system — with cloud plans starting at €59/month and self-hosting completely free with no feature gates. What Is LangWatch? (The LLM Observability Platform Explained) LangWatch is an open-source LLMOps platform that combines production monitoring, automated evaluation, agent simulation testing, and prompt optimization in a single unified system. Founded to address the fragmented tooling problem facing AI teams — where developers typically need 3–5 separate tools for tracing, evals, prompt management, and cost control — LangWatch consolidates all these workflows under one roof. As of 2026, the platform has surpassed 3,000 GitHub stars and supports 10+ LLM providers including OpenAI, Azure, AWS Bedrock, Google Gemini, Deepseek, Groq, MistralAI, VertexAI, and LiteLLM. The platform is built natively on OpenTelemetry, meaning enterprise teams can integrate with existing observability stacks without vendor lock-in. The LLM observability market it operates in is expanding fast: from $1.97 billion in 2025, it’s projected to hit $2.69 billion in 2026 at a 36.3% CAGR, and $9.26 billion by 2030. LangWatch positions itself as the platform for developers who want production-grade AI monitoring without stitching together half a dozen point solutions. ...