How to Build AI Evals from Real Workflow Data

Build AI evals from workflow data by reconstructing a case, identifying what was known at the decision point and agreeing what the agent should do. Turn that expectation into checks for required actions, forbidden actions and outcomes. Review the case before adapting it to your test runner.

Raw records are a starting point. A successful historical outcome might conceal a workaround, missing approval or information that only arrived later. Copying the original response into an “expected answer” field can turn those problems into the behavior the test rewards.

This guide follows a fictional replacement case from evidence to a draft AI test. The downloadable fixture illustrates the content of a case; it is not a FieldSignal export schema or an executable test for a particular framework.

Separate the record from the rule

The customer needs a replacement tomorrow. The original part is unavailable. A specialist proposes an alternative. The inventory record shows stock but does not establish compatibility.

The specialist explains that the deadline prompted the search for an alternative. The process owner establishes the intended rule: verify compatibility before offering a substitute, and obtain approval when the applicable price rule requires it.

That distinction changes the eval. The expected behavior is not “repeat the historical substitution.” It is “follow the approved rule under the information available in this case.”

Keep the source references and rule approval outside the agent’s task input when they contain grading information the agent should not see. A fixture needs context for the system being tested and separate evidence for the people assessing it.

Use only information available at the decision

Choose the decision point: the agent has found an in-stock alternative but has not verified compatibility.

Record the facts the agent may use. For this example, those include the request, deadline, stock result and applicable policy. Do not add a later installation result. That would answer a question the agent was supposed to resolve with incomplete information.

Use an explicit unknown value when a fact is missing. Do not turn “no compatibility result in the available evidence” into “the parts are incompatible.” The first calls for review; the second may justify a different response.

This is also where you record limits. If the source cannot establish the historical state, the case may still be useful for a constructed scenario, but it should not be described as an exact replay.

Define behavior before writing the grader

The fictional rule produces a simple blocked-path case:

Avoid a rigid expected sentence when multiple explanations are acceptable. At the same time, do not replace a consequential action check with a broad score for “helpfulness.”

Use deterministic checks where the test environment exposes the state clearly. A human reviewer or a calibrated rubric can assess explanation quality. If a judge says the answer is good but the tool trace shows an unauthorized order, the operational requirement has failed.

The example fixture separates agent input, evaluator references and expectations. It has no run result. A developer must map those expectations to the actual action names and state in the chosen runner.

Add a permitted path and meaningful counterexamples

A suite containing only blocked cases can reward an agent that always escalates. Add a separate case where compatibility is verified and all applicable approval requirements are satisfied. The agent should be able to progress when the rule permits it.

Then vary a condition with a clear reason: required approval absent, conflicting product information or a timeout during order creation. Preserve the business rule and identify which variable the case changes.

Distinguish three kinds of case in the dataset:

• Observed: based on a source-supported historical instance.
• Adapted: derived from a real case with a changed condition.
• Constructed: designed to test a rule or failure condition without claiming it occurred.

Synthetic cases are useful for deliberate coverage. They do not establish how often the condition occurs in production. Historical cases are useful evidence, but their observed actions are not automatically correct.

Keep some cases separate from implementation tuning. Group near-duplicates and related cases when splitting the dataset so one version does not give away the answer to another.

Review the case and connect it to development

Ask the process owner to approve the intended behavior. Ask a developer to check that the case is executable in the planned environment. Missing mock behavior, hidden state or inconsistent grading can make a valid agent look wrong.

Run a known acceptable behavior and an intentionally unacceptable behavior through the checks before relying on the eval. Record the agent version, policy version, dataset version and relevant environment configuration when comparing changes. For variable behavior, examine repeat trials and case-level failures rather than treating a single pass as stable performance.

Anthropic’s eval guidance discusses clear task specifications and isolated trial environments; those are useful execution checks once the workflow expectation is sound. Engineering reference

FieldSignal makes workflow evidence available through MCP and CLI and supplies Claude and Codex skills to help turn it into draft AI tests. Have your team check the expected action, adapt the case to its test tools and run the test.

Explore the FieldSignal platform if your deployment team needs a repeatable way to bring customer workflow evidence into this process.

Related reading: use MCP and CLI with Claude and Codex, automation acceptance testing, and the complete replacement example.

---