How to Evaluate an AI Product Research Agent
Freeze sourcing evidence, blind-review decisions, and replay failures before using an AI product research agent to order stock.
Evaluate an AI product research agent on sourcing decisions whose evidence you can reconstruct. Freeze what was knowable at the time, ask the agent to cite the records behind its decision, and have a reviewer grade the answer without seeing who wrote it. Use a separate small pilot to learn whether the decision works in your store. This guide adapts Scott's original workflow for building and evaluating sourcing agents. Original SCTCamp guide in Chinese
What goes in the test case?
Start with one decision, such as whether to request a sample or hold a product while a supplier quote is incomplete. Save a task brief with the marketplace, target buyer, inventory ceiling, exclusions, and evidence cutoff. Keep each evidence snapshot and its observation date so someone can replay the task later.
The table below is a fictional evaluation fixture based on the original guide's closet-bin example. Use it to check how an agent handles incomplete and late evidence.
| Field | Fixture input | What the reviewer should check |
|---|---|---|
| Decision | Whether to advance a collapsible closet-bin supplier to sampling for a US seller | The answer states whether sampling is worth pursuing |
| Constraint | First inventory order capped at 500 units, with no new mold | The answer treats a 2,000-unit inventory minimum as a blocker unless the supplier can change it |
| E1 | September 28 snapshot of a competitor listing and dimensions | The answer distinguishes listed dimensions from measured fit |
| E2 | September 25 review excerpt saying a bin did not fit | The answer leaves room for shelf clearance, bin capacity, and assembly as competing explanations |
| E3 | October 2 shipping quote, after the September 30 cutoff | The answer leaves landed cost unresolved for this run |
A reasonable answer is to ask whether the supplier can meet the inventory cap before paying for a sample, and to request packed dimensions for a freight quote. Other answers can pass if they respect the constraints and explain their next step using eligible evidence. The fixture tests reasoning and record handling. For the data collection step, start with the companion evidence-sheet guide. Product research evidence sheet
Which work belongs to code, the model, and the buyer?
Use code for product deduplication, unit conversions, fee formulas, and checking that evidence IDs exist and fall within the cutoff. Give the model judgment tasks that benefit from context, such as proposing explanations for an ambiguous complaint and finding counterexamples. The buyer decides how much inventory risk to take and whether to approve a supplier order.
Give the agent a way to read a record, request more evidence within a set budget, and submit a decision. Its output should include the proposed decision, supporting and conflicting evidence, open questions, and the result that would change its mind. A failed lookup should leave the field unknown. When the evidence or tool budget runs out, the agent can stop with the question still open. This lets you test an agent against a simpler fixed workflow on the same tasks. Anthropic on workflows and agents
How do you run a fair blind review?
- Give the human operator and the agent the same brief, candidates, and frozen evidence. Have them work separately. Flag any case where the operator already knows the outcome.
- Check constraints and evidence IDs in code. A persuasive answer that cites a post-cutoff quote fails the cutoff check.
- Put both answers into the same format, remove identities, and randomize the order. Ask the reviewer to identify the strongest supported claim, the largest missing risk, and whether the next step can be performed. Allow a tie or two failing answers.
- If a model helps grade, swap answer order and compare a sample against human grades. A ranking that flips with order needs review. Run important cases more than once and inspect the evidence and tool calls behind the score. Anthropic's agent evaluation guide · MT-Bench judge study
What should you change after a failure?
Trace the error to the layer that introduced it. A repeated review points to deduplication. An incorrect margin points to the formula or input units. A prohibited inventory order may point to a missing task constraint. If the relevant facts were present and the agent still ignored them, review the prompt, examples, or model.
Keep development cases for tuning, regression cases for past failures, and untouched cases for a final check. Group variants of the same product and copies of the same source before splitting. For decisions about what would have been knowable on an earlier date, freeze each case at that date. If you tune against a held-out case, move it into development and choose a fresh final check. Kapoor and Narayanan on data leakage
Change one component at a time. Replay the same cases and compare decision quality, failed runs, tool cost, and elapsed time. A claim that the agent improved should point to the changed setting and the cases that demonstrate the difference.
When is the workflow ready for a store pilot?
Once the workflow can handle unfamiliar cases and unresolved evidence, choose one sourcing decision with a spending limit, review date, and clear stopping rule. Keep the actual offer, creative, traffic, and fulfillment conditions with the result. Track discarded candidates as well as products you sampled. Blind review can show whether the agent handles evidence and constraints well. A store pilot is where you learn whether shoppers buy at the proposed price and whether you can fulfill profitably. Open the public product research demo