> ## Documentation Index
> Fetch the complete documentation index at: https://docs.moda.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Prompt experiments

> Compare prompt versions over replayed scenarios with moda prompts ab, and generate revised candidates with moda prompts propose.

Before promoting a prompt change, you can measure it. `moda prompts ab` replays a set of scenario cases against a baseline and a candidate prompt and reports which one passes more cases. `moda prompts propose` turns a run's failures into a revised candidate. The dashboard Playground runs the same comparisons interactively.

## Prerequisites

* `@moda-ai/cli` installed and an API key configured (`MODA_API_KEY` or `moda init`).
* A tenant ID, stored by `moda init` or passed as `--tenant-id=<id>`.
* Synced prompts. `moda prompts ab` reads prompt content from disk, but the version IDs attached to each arm come from `.moda/prompts.lock.json` — with an unsynced prompt they are empty and results are not attributed to versions. Pass `--sync=true` to sync before the run. `moda prompts propose` revises a version in the registry, so its prompt must be synced.

## Compare two prompts: moda prompts ab

```bash theme={"dark"}
moda prompts ab --baseline=support.triage --candidate=prompts/support/triage-v2.prompt.md
```

`--baseline` and `--candidate` each accept the key of a discovered prompt or a path to a prompt file (aliases: `--baseline-key`, `--baseline-file`, `--candidate-key`, `--candidate-file`). The baseline runs as the comparison's production arm, the candidate as the proposed arm. Each case is replayed `--seeds` times per arm (default 3, maximum 10) and scored pass/fail.

The command enqueues the run, polls every 15 seconds, and prints a verdict when the run completes:

```text theme={"dark"}
Prompt A/B replay verdict
-------------------------
Replay set: <replay_set_id>
Run id:     <run_id>
Status:     completed
Verdict:    candidate wins; baseline 3/5 (60%), candidate 4/5 (80%)

Per-case:
  <case_id>: baseline=pass candidate=pass
  <case_id>: baseline=fail candidate=pass
```

In JSON and agent output modes, the payload carries `replaySetId`, `runId`, `status`, `verdict`, a `run` object (`caseCount`, `prodPassCount`, `proposedPassCount`, `prodPassRate`, `proposedPassRate`, `notes`), and per-case rows (`caseId`, `prodPass`, `proposedPass`, `prodScore`, `proposedScore`). A `null` pass value means the judge abstained on that case.

### Win criteria

The candidate wins only when it passes strictly more cases than the baseline on a completed run. A tie, a regression, a skipped run, or an errored run is never a candidate win.

### Choosing the replay set

By default the command auto-generates a replay set from recent conversations:

```bash theme={"dark"}
moda prompts ab --baseline=support.triage --candidate=prompts/support/triage-v2.prompt.md \
  --cases=8 --lookback-days=14 --name="Triage rewrite"
```

Two alternatives:

* `--set-id=<replay_set_id>` — reuse an existing replay set (also lets successive experiments run against identical cases).
* `--conversations=<id1>,<id2>` — build a set with one case per conversation, with the scenario seeded from each conversation's first user message.

### Flags

| Flag                      | Default          | Description                                                 |
| ------------------------- | ---------------- | ----------------------------------------------------------- |
| `--baseline=<file\|key>`  | required         | Baseline prompt (production arm).                           |
| `--candidate=<file\|key>` | required         | Candidate prompt (proposed arm).                            |
| `--set-id=<id>`           | —                | Reuse an existing replay set.                               |
| `--conversations=<ids>`   | —                | Comma-separated conversation IDs to build cases from.       |
| `--cases=<n>`             | 5                | Cases to auto-generate (max 500).                           |
| `--lookback-days=<n>`     | 30               | Conversation window for auto-generation (max 365).          |
| `--name=<text>`           | dated name       | Name for a newly created replay set.                        |
| `--seeds=<n>`             | 3                | Replays per case per arm (max 10).                          |
| `--model=<id>`            | —                | Override the assistant model used during replay.            |
| `--sync=true`             | off              | Run `moda prompts sync` before the comparison.              |
| `--no-wait`               | off              | Return after enqueueing instead of polling for the verdict. |
| `--timeout=<ms>`          | 7200000          | Maximum wait for the run to finish (max 24 h).              |
| `--poll-interval=<ms>`    | 15000            | Poll interval while waiting (max 120000).                   |
| `--tenant-id=<id>`        | from `moda init` | Tenant to run against.                                      |

## Generate a revised candidate: moda prompts propose

`moda prompts propose` reads the failing cases of a completed comparison's candidate (proposed) arm and generates a revised prompt. The revision starts from the prompt's `prod`-labeled version, falling back to the latest synced version. The result is registered as a new **unlabeled** version — it is never promoted automatically.

```bash theme={"dark"}
moda prompts propose support.triage \
  --from-run=<run_id> \
  --set-id=<replay_set_id> \
  --out=prompts/support/triage-candidate.prompt.md
```

`--from-run` is the `runId` and `--set-id` the `replaySetId` printed by `moda prompts ab`. The command waits up to 5 minutes for the revision.

The failing cases are split into a repair set and a holdout set; the revision model only ever sees dossiers from the repair cases, and the holdout case IDs are printed so you can judge the candidate on cases it was not tuned on. Result `status` values:

| Status                        | Meaning                                                              |
| ----------------------------- | -------------------------------------------------------------------- |
| `proposed`                    | A candidate was generated and registered as an unlabeled version.    |
| `no_failures`                 | The referenced run has no failing candidate-arm cases to learn from. |
| `revision_unusable`           | The generated revision was not usable as a prompt.                   |
| `revision_rejected_injection` | The revision was rejected by prompt-injection screening.             |

The response also includes the new `versionId`, a `changelog`, `risks`, the repair and holdout case IDs, and a low-sample warning when the run had fewer than two failures.

### Propose flags

| Flag                 | Default    | Description                                                               |
| -------------------- | ---------- | ------------------------------------------------------------------------- |
| `--prompt-key=<key>` | positional | Prompt to revise (or pass the key as the first argument).                 |
| `--from-run=<id>`    | required   | Completed comparison run to learn from.                                   |
| `--set-id=<id>`      | required   | Replay set the run belongs to.                                            |
| `--max-dossiers=<n>` | 8          | Failure dossiers given to the revision model (max 16).                    |
| `--model=<slug>`     | —          | Override the revision model.                                              |
| `--out=<path>`       | —          | Write the candidate content to a file.                                    |
| `--gate`             | off        | Automatically A/B the candidate against the baseline after generating it. |
| `--promote-on-win`   | off        | With `--gate`: promote the candidate to `prod` only on a strict win.      |

`--gate` also accepts the A/B flags `--seeds`, `--assistant-model`, `--gate-timeout` (ms), `--poll-interval` (ms), and `--tenant-id`.

### Gating and promoting a candidate

With `--gate`, the command immediately runs a comparison of the fresh candidate against the baseline prompt (resolved by key from your discovered prompt files) over the same replay set and prints the verdict. Add `--promote-on-win` to promote the candidate to `prod` when — and only when — it passes strictly more cases:

```bash theme={"dark"}
moda prompts propose support.triage \
  --from-run=<run_id> --set-id=<replay_set_id> \
  --gate --promote-on-win
```

<Note>
  The gate scores the full replay set. The holdout cases cannot be isolated into their own run, so a gate win does not by itself rule out overfitting to the repair cases. For a stricter check, gate manually on a fresh set.
</Note>

To gate manually, compare the written `--out` file against the baseline on newly generated cases, then promote if it wins:

```bash theme={"dark"}
moda prompts ab --baseline-key=support.triage \
  --candidate=prompts/support/triage-candidate.prompt.md --auto-generate

moda prompts promote support.triage --label=prod --version=<new_version_id>
```

<Warning>
  Pass the candidate as the `--out` file path, not as the prompt key. The candidate exists only as an unlabeled registry version; `--candidate-key=support.triage` resolves to the baseline content on disk and compares the prompt against itself.
</Warning>

## Interactive comparisons: the Playground

For hands-on iteration, open **Dashboard → Playground** (in the Evaluate group), or click **Open in Playground** on a prompt's detail page. The Playground compares the prompt's production content against a proposed edit over scenario or eval-set datasets, and lets you save an edit as a proposal for review.

Everything connects back to the prompt's **Optimize** tab: proposals and AI-generated candidates queue there for review, comparison runs appear in its run history, and merged candidates become unlabeled versions that you promote from the dashboard or with `moda prompts promote`.

## Next steps

* [Prompt workflow](/prompt-management/workflow) — sync and promote, including the lockfile the experiment arms read.
* [Prompt attribution](/prompt-management/attribution) — link production calls to versions so comparisons reflect real usage.
* [Evaluate](/dashboard/evaluate) — Replays and the Playground in the dashboard.
* [CLI reference](/cli/reference) — output modes and exit codes for `moda prompts` commands.