Why Traditional Software Testing Breaks for Multi-Step AI Agents

Traditional software testing assumes deterministic, predictable behavior: given input X, the function reliably returns Y. Multi-step AI agents violate this assumption at every layer. An agent doesn’t just map inputs to outputs — it perceives context, selects tools, interprets intermediate results, adjusts its plan, and eventually produces an answer through a sequence of decisions that can vary on every run. As of 2026, 79% of organizations have adopted AI agents to some extent, and 57% already have agents in production (Multimodal.dev). Yet over 40% of agentic AI projects are at risk of cancellation by 2027 if governance, observability, and ROI clarity are not established (Gartner). The root cause is almost always testing inadequacy — teams apply unit-test thinking to systems that require trajectory evaluation. A unit test catches a wrong return value; what it cannot catch is an agent that reaches the right answer through a broken series of tool calls that would fail at scale or under edge-case inputs.

What Makes Agent Testing Different?

Multi-step agents introduce three failure modes that traditional tests cannot observe: step repetition (the agent loops on the same action without progress), reasoning-action mismatch (the agent’s stated reasoning diverges from its actual tool call), and cascading tool errors (a malformed argument at step 2 silently corrupts every downstream step). An arXiv 2025 study found that 17.14% of multi-step agent failures are step repetitions and 13.98% are reasoning-action mismatches — making these categories far more common than simple wrong-answer failures.

The 5-Layer Agent Evaluation Framework

A complete agent evaluation framework is built in five layers, each measuring a different level of agent behavior, from the lowest-level tool mechanics to business-outcome alignment. The five layers are: (1) Tool Correctness — did the agent call the right tool with valid arguments; (2) Step Quality — is each individual step’s output accurate and appropriate; (3) Trajectory Coherence — does the sequence of steps form a logical, efficient path to the goal; (4) Task Completion — did the agent achieve the user’s stated goal; and (5) Business Outcome — did task completion translate to real business value (cost saved, error reduced, user time recovered). Most teams in 2025 evaluated only layer 4 (task completion), which is why agent reliability remained poor. By 2026, leading teams instrument all five layers. Without layer 1 (tool correctness), a passing task-completion score can hide systematic tool misuse that will break at scale. Without layer 5 (business outcome), you optimize for proxy metrics that diverge from what actually matters.

Layer 1 — Tool Correctness

Tool misuse is the most common agent failure mode in production: a malformed argument at step 2 silently corrupts every downstream step (Latitude.so). Layer 1 evaluation checks three sub-properties: schema validity (do tool arguments match the expected JSON schema), semantic correctness (are the argument values contextually appropriate, not just structurally valid), and tool selection appropriateness (did the agent choose the right tool for the sub-task rather than an adjacent tool that happens to accept the call).

Layer 3 — Trajectory Coherence

Trajectory coherence measures whether the full sequence of steps is logically consistent and efficient. Scoring approaches include step-count ratio (actual steps / minimum required steps, where values above 1.5 signal inefficiency), backtrack detection (penalize cases where the agent revisits a tool with the same arguments after receiving a valid result), and coherence scoring via LLM-as-judge, where a separate model evaluates whether each step’s inputs and outputs form a sensible chain of reasoning.

Core Metrics — Trajectory vs. Outcome Evaluation

Agent evaluation metrics fall into two temporal categories: outcome metrics measure quality at task completion, while trajectory metrics measure quality at each intermediate step. Both are required for a production-grade evaluation suite. Key trajectory metrics include tool-call accuracy (fraction of calls with valid arguments), step precision (fraction of steps that contribute to goal completion, excluding unnecessary detours), reasoning-action alignment (fraction of steps where the model’s stated rationale matches its actual tool selection), and agentic loop detection (binary flag when step repetition exceeds a threshold, typically 2 identical calls in sequence). Key outcome metrics include task success rate (binary pass/fail on whether the final result satisfies the goal), goal adherence score (0-1 LLM-judge rating of how closely the final output matches the original intent), and latency-to-correct (time elapsed before the agent reaches a valid final state, useful for comparing agent versions). A critical insight from Braintrust’s 2026 evaluation research: trajectory scoring catches regressions that outcome scoring misses because an agent can succeed at the final step while taking a path that would fail under slightly different inputs. Always run both.

The Calibration Problem With LLM-as-Judge

LLM-as-judge achieves approximately 80% agreement with human evaluators at 500x-5000x lower cost — but drops to 60-70% agreement in expert domains like legal, medical, or specialized code review (Galileo). The calibration fix is domain-specific rubrics: instead of asking the judge “is this a good answer,” provide a 5-point rubric with concrete behavioral anchors for each score level. Without rubrics, judge agreement on subjective quality scores varies by as much as 30% across different prompts for the same underlying content.

Building Your Golden Dataset (Size, Sources, and Curation)

A golden dataset is the foundation of reproducible agent evaluation — a curated set of test cases with inputs, expected tool sequences, and acceptable final outputs used to measure agent quality across releases. Dataset size should be calibrated to risk level: 50 test cases for proof-of-concept validation, 100-200 for production deployment, and 300-500+ for mission-critical agents in high-stakes domains like finance, healthcare, or legal (Confident AI). Each test case should include four components: (1) a realistic user input derived from actual or representative usage, (2) the ground-truth tool sequence (which tools should be called, in what order, with what arguments), (3) acceptable final-output criteria (either exact match, semantic similarity threshold, or LLM-judge rubric), and (4) a failure mode annotation describing the failure the case is designed to catch.

Sourcing Test Cases

The most effective golden datasets are built from three sources in combination. Production logs (with PII removed) surface real-world edge cases that synthetic generation misses — mine your top 20 failure cases and convert them to test cases. Adversarial generation uses a separate LLM to generate inputs designed to trigger known failure modes: ambiguous requests, conflicting constraints, and inputs that look like simpler tasks but require multi-step reasoning. Domain expert authoring — having a subject-matter expert write 20-30 cases — provides coverage of business-specific edge cases that neither logs nor synthetic generation can produce. AWS’s Strands Evals framework goes further: its ExperimentGenerator automatically generates test cases and rubrics from high-level task descriptions using LLMs, reducing manual curation effort by an estimated 60-70% for initial dataset construction.

Dataset Maintenance Cadence

Golden datasets decay. As the agent evolves, test cases from six months ago may no longer reflect the agent’s current capability scope or the product’s current use cases. Treat the dataset like a test suite in a traditional codebase: run a quarterly review, retire cases that no longer reflect real usage patterns, and add at least 10 new cases per sprint from production failure analysis. Track dataset coverage metrics: what fraction of your agent’s tools, task categories, and failure modes are represented by at least 5 test cases.

LLM-as-Judge vs. Human-in-the-Loop — When to Use Each

LLM-as-judge and human review are complementary, not competing, evaluation strategies. The right hybrid depends on risk tier, not team preference. LLM-as-judge achieves approximately 80% agreement with human evaluators at 500x-5000x lower cost in general-purpose domains, but drops to 60-70% agreement in expert domains like legal analysis, medical triage, or specialized code review (Galileo, 2026). This means LLM-as-judge is appropriate for high-volume, lower-risk evaluation: routing quality checks, format compliance, step-count efficiency, tool schema validation, and regression testing in CI/CD pipelines. At that cost differential, it is the only economically viable approach for evaluating every deployment. Human-in-the-loop is appropriate for low-volume, high-stakes decisions: safety evaluation, content moderation in sensitive domains, final-output quality for mission-critical agents, and calibrating new LLM-judge rubrics. The practical takeaway: never rely on a single approach — build a hybrid escalation pipeline that routes borderline cases from LLM-as-judge to human review automatically.

The Hybrid Escalation Pattern

The most effective teams use a two-stage evaluation pipeline: LLM-as-judge runs on every test case and flags outliers (cases scoring below 0.6 or above 0.95 deserve human review — the low end for failure analysis, the high end to verify that easy cases aren’t inflating pass rates). Human reviewers handle only flagged cases, typically 5-15% of the total set. This hybrid approach achieves near-human evaluation quality at a cost roughly equivalent to running LLM-as-judge on 100% plus a 10% human overhead. For mission-critical agents in expert domains (legal research, medical triage, financial analysis), raise the LLM-judge threshold to 0.75 before escalating to human review and plan for 20-30% human oversight.

Integrating Agent Evals into Your CI/CD Pipeline

CI/CD integration transforms evaluation from a periodic manual activity into an automated deployment gate — blocking releases when agent quality regresses below a defined threshold. The standard pattern uses pytest with DeepEval for local and CI evaluation, with a 90% pass-rate threshold as the deployment gate. DeepEval is pytest-native, making adoption straightforward for existing Python test pipelines; it is used by OpenAI, Google, and Microsoft and provides 50+ research-backed metrics for agents, chatbots, RAG, single-turn, multi-turn, and safety evaluation.

# conftest.py — DeepEval CI gate
import pytest
from deepeval import assert_test
from deepeval.metrics import TaskCompletionMetric, ToolCorrectnessMetric
from deepeval.test_case import LLMTestCase, ToolCall

@pytest.mark.parametrize("test_case", golden_dataset)
def test_agent_trajectory(test_case):
    actual = run_agent(test_case["input"])
    case = LLMTestCase(
        input=test_case["input"],
        actual_output=actual["output"],
        tools_called=[ToolCall(**t) for t in actual["tool_calls"]],
        expected_tools=[ToolCall(**t) for t in test_case["expected_tools"]],
    )
    assert_test(case, [
        ToolCorrectnessMetric(threshold=0.9),
        TaskCompletionMetric(threshold=0.85),
    ])

GitHub Actions Deployment Gate

# .github/workflows/agent-eval.yml
name: Agent Evaluation Gate
on: [pull_request]
jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run agent evals
        run: |
          pip install deepeval
          deepeval test run tests/agent/ --min-success-rate 0.90
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
      - name: Upload eval results
        uses: actions/upload-artifact@v4
        with:
          name: eval-report
          path: deepeval_results.json

The 90% pass-rate threshold is a starting point — adjust down to 85% for experimental agents and up to 95% for production agents handling PII or financial decisions. Pair DeepEval for CI enforcement with Braintrust for production traceability: Braintrust (which raised $80M at an $800M valuation in February 2026) provides the experiment-level logging and A/B comparison that CI tools lack.

Catching the Silent Killers — Tool-Call Failures and Cascade Errors

Tool-call failures are the silent killers of multi-step agent reliability. Unlike an exception that immediately halts execution, a tool call with subtly wrong arguments often succeeds at the API level while returning garbage data that the agent continues to reason over. By the time the final output is wrong, the root cause — a malformed query argument at step 2 — is buried under several subsequent steps of plausible-looking reasoning. Detecting cascade errors requires step-level instrumentation: log every tool call with its full input arguments, the raw API response, and the agent’s next action, then run tool-output validation at each step rather than only at task completion. Key detection signals include: schema deviation (any argument that fails JSON Schema validation, even for non-required fields), empty or null results from tools that should return data (agents frequently continue past null returns without flagging them), and reasoning pivot without cause (the agent changes strategy without receiving new information — a strong signal that a tool returned unexpected data).

The Step-2 Problem

The Step-2 Problem describes a specific failure pattern: the agent correctly identifies the first tool to call and calls it correctly, but uses the result to parameterize the second tool call with a structurally valid but semantically wrong argument. Because the second call succeeds (it doesn’t throw an error), the agent treats its output as ground truth and compounds the error through every subsequent step. Detection requires semantic validation of tool outputs — checking not just that the result matches the expected JSON schema, but that the values are contextually coherent with the original task. For example, a search tool returning zero results for a query that should return many is a signal worth catching at step 2, not at step 8 when the agent produces a summary based on an empty search result set.

Top Agent Evaluation Frameworks Compared (2026)

DeepEval wins for teams that want to embed evaluation inside their existing pytest workflow — it requires no new infrastructure and adds evaluation as first-class test assertions. Braintrust wins for teams that need experiment-level comparison (comparing agent v1.2 to v1.3 across the same dataset) and production trace logging. LangSmith is the obvious choice if the team already uses LangChain/LangGraph; it provides native trace capture with minimal setup. Arize Phoenix focuses on observability and drift detection — useful for production monitoring but less suitable as a primary CI evaluation tool. AWS Strands Evals makes the most sense for teams already on Bedrock; its ExperimentGenerator is the standout differentiator for teams that want automated test case generation. Galileo targets enterprise teams that need policy-level quality guardrails with compliance reporting.

Mixing Tools Is Correct

The mature 2026 pattern is not “pick one framework” but “use two in tandem.” The most common combination is DeepEval for CI gate enforcement (where pytest integration and pass-rate thresholds are the priority) paired with Braintrust for production monitoring (where experiment logging, A/B comparison, and human review interfaces are the priority). Amazon Bedrock AgentCore Evaluations, which became GA on March 31, 2026, is emerging as a third option for AWS-native teams that want fully managed evaluation infrastructure.

Production Monitoring and Continuous Feedback Loops

Pre-deployment evaluation catches known failure modes; production monitoring catches the unknown ones. The two temporal dimensions — pre-deployment validation and continuous production monitoring — are both required for production-grade agents, yet most teams in 2025 ran only the former. According to Adaline’s 2026 evaluation methodology report, teams that add continuous production monitoring reduce mean-time-to-detect agent regressions from days to hours. Production monitoring requires three capabilities: trace collection (capturing every tool call, argument, result, and agent decision in a structured log), anomaly detection (alerting when tool error rates, latency, or outcome scores deviate from baseline by more than one standard deviation), and feedback integration (routing user thumbs-down signals and human review flags back into the evaluation pipeline as new test cases). OpenTelemetry has emerged as the standard trace format for agent observability in 2026; Strands Evals, Arize Phoenix, and Braintrust all support OTel ingestion natively, making trace data portable across tools without vendor lock-in. The net effect: production monitoring transforms evaluation from a one-time gate into a continuous quality signal.

Connecting Production Failures to Test Cases

Every production failure should become a test case. Establish a triage process: when monitoring detects an anomaly or a user reports a failure, a team member reviews the trace, confirms it represents a genuine failure mode (not user error), strips PII, and adds it to the golden dataset with a failure-mode annotation. This closed-loop process is what separates teams whose agent quality improves over time from teams whose failure rates stay flat despite ongoing development. Teams running the closed loop consistently report a 30-40% reduction in repeat failure rates within two quarters of implementation.

Building a Complete Agent Testing Checklist

A production-ready agent testing program covers evaluation across the full development lifecycle: pre-launch validation, CI/CD gating, and continuous production monitoring. Use this checklist to assess your current coverage.

Pre-Launch Validation

• Golden dataset built with 100+ cases (300+ for mission-critical)
• Dataset covers all tools the agent can call, with at least 5 cases per tool
• Adversarial test cases included for known failure modes
• Tool-correctness metrics pass at ≥ 90% on golden dataset
• Task-completion rate ≥ 85% on golden dataset
• LLM-judge rubrics written and calibrated against human baselines

CI/CD Integration

• Evaluation runs on every pull request as a deployment gate
• Pass-rate threshold enforced (90% minimum; 95% for PII/financial agents)
• Eval results logged to a tracking system (Braintrust, LangSmith, or equivalent)
• Regression detection: new PR is blocked if pass rate drops > 5% vs. baseline
• Step-level tool-call logging enabled in all test runs

Production Monitoring

• OpenTelemetry trace collection live on all production agent requests
• Anomaly alerts configured for tool error rate, latency, and outcome score
• User feedback signals routed to evaluation pipeline
• Closed-loop process defined: production failure → test case within 48 hours
• Dataset review scheduled quarterly

FAQ

What is AI agent testing and why is it different from traditional software testing?

AI agent testing is the practice of evaluating multi-step AI agents for correctness, reliability, and goal alignment across the full decision trajectory — not just the final output. Unlike traditional software tests that check deterministic function outputs, agent testing must evaluate tool selection, argument quality, step sequence coherence, and final task completion across non-deterministic, multi-turn interactions. Traditional unit tests cannot detect step repetition, reasoning-action mismatch, or cascading tool errors that are the most common production failure modes for agents.

How large should my golden dataset be for AI agent evaluation?

Dataset size should match risk level: 50 test cases for proof-of-concept validation, 100-200 for standard production agents, and 300-500+ for mission-critical agents in high-stakes domains like finance, healthcare, or legal. Each case needs four components: a realistic input, an expected tool sequence, acceptable output criteria, and a failure-mode annotation. Build from production logs, adversarial generation, and domain-expert authoring for the best coverage.

What is LLM-as-judge evaluation and how accurate is it?

LLM-as-judge evaluation uses a separate language model to assess the quality of another model’s output, using structured rubrics to produce numeric scores. It achieves approximately 80% agreement with human evaluators in general domains at 500x-5000x lower cost than human review. Accuracy drops to 60-70% in expert domains (legal, medical, specialized code). Calibrate with domain-specific rubrics and a hybrid escalation pattern: LLM-as-judge handles 85-95% of cases, human review handles flagged outliers.

Which agent evaluation framework should I use in 2026?

The answer depends on your stack and priorities. DeepEval is best for pytest-native CI evaluation with 50+ metrics. Braintrust is best for production traceability and A/B experiment comparison. LangSmith is the default choice for LangChain/LangGraph teams. AWS Strands Evals is best for Bedrock teams that want automated test case generation. The mature 2026 pattern is DeepEval for CI gates paired with Braintrust or LangSmith for production monitoring — using two tools in tandem rather than relying on a single framework.

How do I integrate AI agent evaluation into my CI/CD pipeline?

Use a pytest-native framework like DeepEval to write agent evaluation as standard test assertions with a minimum pass-rate threshold (90% is the standard starting point). Add a GitHub Actions workflow that runs evals on every pull request and blocks merges when the pass rate falls below the threshold. Pair this with a production tracing tool (Braintrust or LangSmith) to log experiment results across releases. Amazon Bedrock AgentCore Evaluations (GA since March 2026) provides a fully managed alternative for AWS-native teams.