Evaluating AI Agents: Building Test Suites for Non-Deterministic Systems

Why Traditional Tests Fail When Evaluating AI Agents (and What to Do Instead)

As a Senior Software Engineer with a decade of experience across Frontend, Web3, and now AI, I've seen my share of testing methodologies. From meticulous unit tests for a React component to integration tests for a smart contract, I thought I had a handle on what "good testing" looked like. Then I started building AI agents.

Let me tell you, if you're trying to apply traditional unit testing to an AI agent, you're going to have a bad time. The very nature of these systems – their non-determinism, their reliance on complex LLM interactions, and their ability to generalize – shatters the deterministic, input-output paradigm that traditional tests thrive on.

"But Amit," you might say, "can't I just assert that given input X, the agent always returns output Y?" The short answer is: no, not reliably. The long answer is: while you can test specific, isolated functions within your agent, you cannot effectively test the holistic agent behavior this way. Here's why:

1. Non-Determinism: LLMs are inherently non-deterministic. Even with the same prompt and temperature=0, you might get subtle variations in output. Asserting exact string matches becomes a game of whack-a-mole.
2. Context Dependence: An agent's response often depends on the entire conversation history or a complex internal state, not just the immediate input.
3. Emergent Behavior: The true power of agents lies in their emergent capabilities, their ability to adapt and reason. Testing for this requires evaluating the quality of the "reasoning," not just the final verbatim output.
4. Cost: Running an entire agent interaction, especially one involving multiple tools or API calls, can be slow and expensive. You can't run thousands of these like typical unit tests before every commit.

So, how do we build confidence in these sophisticated, unpredictable systems? Over the past year, I've evolved an evaluation harness that actually works in production, leveraging what I call "golden traces," rubric grading, and robust regression catching.

The Production-Ready Evaluation Harness: Golden Traces & Rubric Grading

My evaluation strategy revolves around capturing successful, desired agent interactions and then developing a system to automatically assess how well new agent versions align with these "golden standards."

1. Golden Traces: Your Agent's "Happy Path" Snapshots

A golden trace is essentially a recording of a perfect (or near-perfect) interaction sequence with your agent. This includes:

• User Input: The exact prompt the user provided.
• Agent's Thought Process (if exposed): The internal monologue, tool calls, and observations your agent made.
• Agent's Final Output: The ultimate response presented to the user.
• Tool Inputs/Outputs: Specific data exchanged with external services or tools.

How to create them:

Initially, when you're developing a new agent feature, manually run through the desired scenarios. If the agent performs well, save that interaction. I usually store these as JSON files or in a database.

// Example of a simplified golden trace structure
interface GoldenTrace {
  id: string;
  scenarioDescription: string;
  userPrompt: string;
  expectedFinalOutputKeywords: string[]; // Keywords to look for in the final output
  expectedToolCalls?: { toolName: string; args: Record<string, any> }[]; // Optional: for strict tool usage
  // ... potentially more detailed internal states
}

const goldenTraces: GoldenTrace[] = [
  {
    id: "gt-001",
    scenarioDescription: "Order status check - success",
    userPrompt: "What's the status of my order AB123?",
    expectedFinalOutputKeywords: ["status is shipped", "order AB123"],
    expectedToolCalls: [{ toolName: "getOrderStatus", args: { orderId: "AB123" } }],
  },
  // ... more traces
];

The key here is variety. Don't just pick easy, perfect examples. Include edge cases, slightly ambiguous prompts, and scenarios where the agent should ask for clarification.

2. Rubric Grading: Beyond Binary Pass/Fail

This is where evaluators shine. Instead of a simple true/false, we define a rubric to grade the agent's performance. My rubrics typically include:

• Correctness: Did the agent answer the question or complete the task accurately?
• Completeness: Was the answer comprehensive, or did it miss crucial information?
• Relevance: Was the agent's output on topic and relevant to the user's intent?
• Clarity/Conciseness: Was the output easy to understand and free from unnecessary verbosity?
• Tool Usage: Did the agent correctly identify and use the appropriate tools with valid arguments? (This is often a binary check, but can be nuanced).
• Safety/Harmfulness: Did the agent exhibit any undesirable behaviors (e.g., hallucinations, unsafe content)?

How to implement:

You can design dedicated LLM-based evaluators using a separate LLM call. This LLM receives the user prompt, the actual agent's trace (including internal thoughts and final output), and sometimes the golden final output. It then grades the agent against the rubric.

// Pseudocode for an LLM-based rubric evaluator
async function evaluateAgentResponse(
  userPrompt: string,
  agentResponse: string, // Full agent output, including thoughts if available
  goldenStandardKeywords: string[],
  // ... other context
): Promise<EvaluationScores> {
  const llmEvaluatorPrompt = `
    You are an AI assistant designed to evaluate the performance of another AI agent.
    Evaluate the agent's response based on the following criteria:

    User Prompt: "${userPrompt}"
    Agent's Final Response: "${agentResponse}"
    Expected Keywords (from golden trace): "${goldenStandardKeywords.join(', ')}"

    --- Rubric ---
    1. Correctness (0-5): How accurately did the agent address the user's prompt? (5=perfect, 0=completely wrong)
    2. Completeness (0-5): Did the agent provide all necessary information?
    3. Relevance (0-5): Was the agent's response on topic?
    4. Tool Usage (Yes/No): Did the agent appropriately use its tools?
    5. Safety (Yes/No): Is the response safe and non-harmful?

    Provide your evaluation in a JSON object format:
    {
      "correctness": score,
      "completeness": score,
      "relevance": score,
      "toolUsage": "Yes" | "No",
      "safety": "Yes" | "No",
      "critique": "A brief explanation of the grades."
    }
  `;

  const evaluationResult = await llm.generate(llmEvaluatorPrompt);
  return JSON.parse(evaluationResult.text);
}

By using an LLM for grading, you get rich, narrative feedback in addition to scores, which is invaluable for debugging.

3. Regression Catching & Thresholds

With a library of golden traces and an LLM-based grading system, you can build a powerful regression suite.

The Process:

1. For each golden trace, run your new agent against it.
2. Capture the agent's full output/trace.
3. Have your LLM evaluator grade the new agent's performance against the trace criteria.
4. Compare these new scores to the "baseline" scores (which you've also run and saved for the current production version).

Mermaid Diagram: Evaluation Flow

graph TD
    A[Start Evaluation] --> B{Select Golden Trace};
    B --> C[Run Agent with Trace Prompt];
    C --> D[Capture Agent Output & Trace];
    D --> E[LLM Evaluator Grades Agent Output];
    E --> F{Compare Scores to Baseline?};
    F -- Yes --> G[Detect Regression];
    F -- No --> H[Record New Scores];
    H --> I{All Traces Evaluated?};
    I -- No --> B;
    I -- Yes --> J[End Evaluation];

Thresholds:

Don't expect every score to be perfect every time. Define acceptable thresholds for each rubric item. For example, if your average "correctness" score for a trace drops by more than 0.5 points compared to the baseline, or if "safety" switches from "Yes" to "No," that's a regression that needs attention.

This approach gives you:

• Early Warning: Catching regressions before they hit production.
• Actionable Feedback: The LLM's critique helps pinpoint why a regression occurred.
• Confidence: A clear signal that your agent is performing at or above baseline.

Practical Tips and Lessons Learned

• Start Small, Iterate Often: Don't try to build the perfect eval harness upfront. Start with 5-10 key golden traces and expand as you go.
• Mock Dependencies: For tools that interact with external APIs, mock them extensively. This makes your evaluations faster, cheaper, and more reproducible. You want to test your agent's reasoning, not the external service's uptime.
• Version Your Traces: As your agent evolves, so too might the expected "golden" behavior. Version your golden traces alongside your code.
• Human-in-the-Loop: While automated, nothing beats a human review. Periodically review a random sample of evaluated traces and their grades to ensure your LLM evaluator is calibrated correctly.
• Cost Management: Running LLM evaluations can be costly. Be mindful of the number of traces and the complexity of the evaluation prompts. Batch calls where possible.
• Clear Prompts for Evaluators: The quality of your LLM-based evaluation depends entirely on the clarity and comprehensiveness of the prompt you provide to the evaluator LLM. Be explicit about the rubric criteria.

Conclusion

Evaluating AI agents is a paradigm shift from traditional software testing. It demands a different mindset, one that embraces non-determinism and focuses on qualitative assessment over strict binary outcomes. By adopting golden traces, rubric-based LLM grading, and a robust regression detection system, you can build a powerful and practical evaluation harness that gives you real confidence in your agent's performance. This isn't just theory; it's the framework I've successfully implemented and refined in production.

I'm always eager to discuss the fascinating world of AI agents and engineering challenges. Connect with me on LinkedIn or X to share your thoughts or ask questions!