Skip to main content
The Evaluate section closes the loop between observing your agent and changing it: Replays holds eval sets distilled from real conversations, the Playground runs Production-vs-Proposed comparisons over them, and Skills (under Improve) turns observed behavior into releasable skill documents.

​
Replays

Open Evaluate → Replays. A replay set (also called an eval set) is a collection of test cases distilled from your production conversations. The page lists your sets with case counts, source, and last-updated time, and highlights the primary set. Ways to create a set:
  • Generate set — auto-generates a set from recent conversations; the new set shows a “Generating” state until it is ready.
  • New set — creates an empty named set you fill yourself.
  • Create eval set on a use case detail page — builds a set from that use case’s conversations and opens it in the Playground.
  • Add to eval set on a task in a conversation’s World State view — turns a single observed task into a case.
Clicking a set opens its detail page: case count, origin, last updated, an editable description, the Latest comparison, and the Cases list. Sets can be deleted from here. Individual replay runs have their own page showing the case, scenario, and resulting transcript.

​
Playground

Open Evaluate → Playground. The Playground is one workspace that runs the same work through two configurations — Production (the live baseline) and Proposed (your change under test) — and compares the results side by side.

​
Choose the input

Two dataset modes:
  • One input — type a scenario (“What should the agent help the user do?”) and optional success criteria. No setup required.
  • Saved eval set — pick a replay set. You can generate additional cases from use cases, add cases from a search over stored conversations, and filter cases with use-case chips.

​
Define the change under test

The Production column pins the live baseline: the prompt and version are read-only, and a Diff toggle renders the live prompt as a line diff against your proposed edit. In the Proposed column you set the change under test:
  • a managed prompt and version,
  • candidate skills.
Each column has its own model selector, so a comparison can also test a model change.

​
Run and read the comparison

  • The primary action is Run comparison (one input) or Compare N cases (eval set); Cmd/Ctrl+Enter also runs it.
  • A pre-run cost estimate appears next to the run button and in the confirmation dialog. It is a rough heuristic based on case count and model pricing, not a quote.
  • A usage meter in the header shows this workspace’s replay and playground inference spend. Runs execute on inference provisioned by Moda — a dedicated inference key is created automatically on your first replay or playground run, and its spend also appears under Settings → Provider credentials.
  • Results render as a per-case Production vs Proposed matrix with judged pass/fail per side. A row inspector drills into a single case, can re-run just that case, and can save case edits.
  • Runs continue on the server if you navigate away; run history lets you reopen a comparison later.
Settings → Provider credentials stores encrypted LLM provider keys for your workspace. OpenAI and Anthropic are the only supported providers.

​
Skills

Open Improve → Skills. Skills are SKILL.md documents distilled from your agent’s observed behavior, reviewed and released from this page. Three tabs:
  • Opportunities — proposed skills. Each proposal can be applied, dismissed, opened in the Playground, or inspected via its evidence links. Dismissing a proposal hides it in your browser only; it does not delete the proposal.
  • Active — the live skill surface: authored and observed skills, with per-skill policy controls.
  • Changes — generation-run history with per-run events, the generated skill payloads, a release diff, a comparison summary, and release approval.

​
Generate skills

The Generate skills button opens a dialog where you pick the conversation source, an optional session limit and time window, and optionally a replay comparison against a chosen eval set. The run appears under Changes while it processes, and an active-run banner tracks progress on the page.

​
Approve a release

In Changes, select a run and use Approve for pull. The approved version becomes the release served by moda skills pull, and the previous production release is deprecated. Pull released skills into a repository with the CLI:

​
Next steps

  • Use Cases — create eval sets scoped to one kind of user work.
  • Prompt experiments — compare prompt versions from the CLI with moda prompts ab.
  • CLI reference — full command list, including moda skills.