> ## Documentation Index
> Fetch the complete documentation index at: https://docs.moda.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Replays, Playground, and Skills

> Build eval sets from production conversations, compare Production vs Proposed configurations in the Playground, and review and release generated skills.

The Evaluate section closes the loop between observing your agent and changing it: **Replays** holds eval sets distilled from real conversations, the **Playground** runs Production-vs-Proposed comparisons over them, and **Skills** (under Improve) turns observed behavior into releasable skill documents.

## Replays

Open **Evaluate → Replays**. A replay set (also called an eval set) is a collection of test cases distilled from your production conversations. The page lists your sets with case counts, source, and last-updated time, and highlights the primary set.

Ways to create a set:

* **Generate set** — auto-generates a set from recent conversations; the new set shows a "Generating" state until it is ready.
* **New set** — creates an empty named set you fill yourself.
* **Create eval set** on a [use case detail page](/dashboard/use-cases) — builds a set from that use case's conversations and opens it in the Playground.
* **Add to eval set** on a task in a conversation's World State view — turns a single observed task into a case.

Clicking a set opens its detail page: case count, origin, last updated, an editable description, the **Latest comparison**, and the **Cases** list. Sets can be deleted from here. Individual replay runs have their own page showing the case, scenario, and resulting transcript.

## Playground

Open **Evaluate → Playground**. The Playground is one workspace that runs the same work through two configurations — **Production** (the live baseline) and **Proposed** (your change under test) — and compares the results side by side.

### Choose the input

Two dataset modes:

* **One input** — type a scenario ("What should the agent help the user do?") and optional success criteria. No setup required.
* **Saved eval set** — pick a replay set. You can generate additional cases from use cases, add cases from a search over stored conversations, and filter cases with use-case chips.

### Define the change under test

The Production column pins the live baseline: the prompt and version are read-only, and a **Diff** toggle renders the live prompt as a line diff against your proposed edit. In the Proposed column you set the change under test:

* a managed prompt and version,
* candidate skills.

Each column has its own model selector, so a comparison can also test a model change.

### Run and read the comparison

* The primary action is **Run comparison** (one input) or **Compare N cases** (eval set); Cmd/Ctrl+Enter also runs it.
* A pre-run **cost estimate** appears next to the run button and in the confirmation dialog. It is a rough heuristic based on case count and model pricing, not a quote.
* A **usage meter** in the header shows this workspace's replay and playground inference spend. Runs execute on inference provisioned by Moda — a dedicated inference key is created automatically on your first replay or playground run, and its spend also appears under **Settings → Provider credentials**.
* Results render as a per-case **Production vs Proposed** matrix with judged pass/fail per side. A row inspector drills into a single case, can re-run just that case, and can save case edits.
* Runs continue on the server if you navigate away; run history lets you reopen a comparison later.

<Note>
  **Settings → Provider credentials** stores encrypted LLM provider keys for your workspace. OpenAI and Anthropic are the only supported providers.
</Note>

## Skills

Open **Improve → Skills**. Skills are SKILL.md documents distilled from your agent's observed behavior, reviewed and released from this page. Three tabs:

* **Opportunities** — proposed skills. Each proposal can be applied, dismissed, opened in the Playground, or inspected via its evidence links. Dismissing a proposal hides it in your browser only; it does not delete the proposal.
* **Active** — the live skill surface: authored and observed skills, with per-skill policy controls.
* **Changes** — generation-run history with per-run events, the generated skill payloads, a release diff, a comparison summary, and release approval.

### Generate skills

The **Generate skills** button opens a dialog where you pick the conversation source, an optional session limit and time window, and optionally a replay comparison against a chosen eval set. The run appears under Changes while it processes, and an active-run banner tracks progress on the page.

### Approve a release

In **Changes**, select a run and use **Approve for pull**. The approved version becomes the release served by `moda skills pull`, and the previous production release is deprecated. Pull released skills into a repository with the CLI:

```bash theme={"dark"}
moda skills pull                    # approved release into .claude/skills (and .cursor/rules if .cursor exists)
moda skills pull --status=proposed  # pull the latest draft (not yet approved) release instead
```

## Next steps

* [Use Cases](/dashboard/use-cases) — create eval sets scoped to one kind of user work.
* [Prompt experiments](/prompt-management/experiments) — compare prompt versions from the CLI with `moda prompts ab`.
* [CLI reference](/cli/reference) — full command list, including `moda skills`.