Prompt experiments

Before promoting a prompt change, you can measure it. moda prompts ab replays a set of scenario cases against a baseline and a candidate prompt and reports which one passes more cases. moda prompts propose turns a run’s failures into a revised candidate. The dashboard Playground runs the same comparisons interactively.

Prerequisites

@moda-ai/cli installed and an API key configured (MODA_API_KEY or moda init).
A tenant ID, stored by moda init or passed as --tenant-id=<id>.
Synced prompts. moda prompts ab reads prompt content from disk, but the version IDs attached to each arm come from .moda/prompts.lock.json — with an unsynced prompt they are empty and results are not attributed to versions. Pass --sync=true to sync before the run. moda prompts propose revises a version in the registry, so its prompt must be synced.

Compare two prompts: moda prompts ab

moda prompts ab --baseline=support.triage --candidate=prompts/support/triage-v2.prompt.md

--baseline and --candidate each accept the key of a discovered prompt or a path to a prompt file (aliases: --baseline-key, --baseline-file, --candidate-key, --candidate-file). The baseline runs as the comparison’s production arm, the candidate as the proposed arm. Each case is replayed --seeds times per arm (default 3, maximum 10) and scored pass/fail. The command enqueues the run, polls every 15 seconds, and prints a verdict when the run completes:

Prompt A/B replay verdict
-------------------------
Replay set: <replay_set_id>
Run id:     <run_id>
Status:     completed
Verdict:    candidate wins; baseline 3/5 (60%), candidate 4/5 (80%)

Per-case:
  <case_id>: baseline=pass candidate=pass
  <case_id>: baseline=fail candidate=pass

In JSON and agent output modes, the payload carries replaySetId, runId, status, verdict, a run object (caseCount, prodPassCount, proposedPassCount, prodPassRate, proposedPassRate, notes), and per-case rows (caseId, prodPass, proposedPass, prodScore, proposedScore). A null pass value means the judge abstained on that case.

Win criteria

The candidate wins only when it passes strictly more cases than the baseline on a completed run. A tie, a regression, a skipped run, or an errored run is never a candidate win.

Choosing the replay set

By default the command auto-generates a replay set from recent conversations:

moda prompts ab --baseline=support.triage --candidate=prompts/support/triage-v2.prompt.md \
  --cases=8 --lookback-days=14 --name="Triage rewrite"

Two alternatives:

--set-id=<replay_set_id> — reuse an existing replay set (also lets successive experiments run against identical cases).
--conversations=<id1>,<id2> — build a set with one case per conversation, with the scenario seeded from each conversation’s first user message.

Flags

Flag	Default	Description
`--baseline=<file\|key>`	required	Baseline prompt (production arm).
`--candidate=<file\|key>`	required	Candidate prompt (proposed arm).
`--set-id=<id>`	—	Reuse an existing replay set.
`--conversations=<ids>`	—	Comma-separated conversation IDs to build cases from.
`--cases=<n>`	5	Cases to auto-generate (max 500).
`--lookback-days=<n>`	30	Conversation window for auto-generation (max 365).
`--name=<text>`	dated name	Name for a newly created replay set.
`--seeds=<n>`	3	Replays per case per arm (max 10).
`--model=<id>`	—	Override the assistant model used during replay.
`--sync=true`	off	Run `moda prompts sync` before the comparison.
`--no-wait`	off	Return after enqueueing instead of polling for the verdict.
`--timeout=<ms>`	7200000	Maximum wait for the run to finish (max 24 h).
`--poll-interval=<ms>`	15000	Poll interval while waiting (max 120000).
`--tenant-id=<id>`	from `moda init`	Tenant to run against.

Generate a revised candidate: moda prompts propose

moda prompts propose reads the failing cases of a completed comparison’s candidate (proposed) arm and generates a revised prompt. The revision starts from the prompt’s prod-labeled version, falling back to the latest synced version. The result is registered as a new unlabeled version — it is never promoted automatically.

moda prompts propose support.triage \
  --from-run=<run_id> \
  --set-id=<replay_set_id> \
  --out=prompts/support/triage-candidate.prompt.md

--from-run is the runId and --set-id the replaySetId printed by moda prompts ab. The command waits up to 5 minutes for the revision. The failing cases are split into a repair set and a holdout set; the revision model only ever sees dossiers from the repair cases, and the holdout case IDs are printed so you can judge the candidate on cases it was not tuned on. Result status values:

Status	Meaning
`proposed`	A candidate was generated and registered as an unlabeled version.
`no_failures`	The referenced run has no failing candidate-arm cases to learn from.
`revision_unusable`	The generated revision was not usable as a prompt.
`revision_rejected_injection`	The revision was rejected by prompt-injection screening.

The response also includes the new versionId, a changelog, risks, the repair and holdout case IDs, and a low-sample warning when the run had fewer than two failures.

Propose flags

Flag	Default	Description
`--prompt-key=<key>`	positional	Prompt to revise (or pass the key as the first argument).
`--from-run=<id>`	required	Completed comparison run to learn from.
`--set-id=<id>`	required	Replay set the run belongs to.
`--max-dossiers=<n>`	8	Failure dossiers given to the revision model (max 16).
`--model=<slug>`	—	Override the revision model.
`--out=<path>`	—	Write the candidate content to a file.
`--gate`	off	Automatically A/B the candidate against the baseline after generating it.
`--promote-on-win`	off	With `--gate`: promote the candidate to `prod` only on a strict win.

--gate also accepts the A/B flags --seeds, --assistant-model, --gate-timeout (ms), --poll-interval (ms), and --tenant-id.

Gating and promoting a candidate

With --gate, the command immediately runs a comparison of the fresh candidate against the baseline prompt (resolved by key from your discovered prompt files) over the same replay set and prints the verdict. Add --promote-on-win to promote the candidate to prod when — and only when — it passes strictly more cases:

moda prompts propose support.triage \
  --from-run=<run_id> --set-id=<replay_set_id> \
  --gate --promote-on-win

The gate scores the full replay set. The holdout cases cannot be isolated into their own run, so a gate win does not by itself rule out overfitting to the repair cases. For a stricter check, gate manually on a fresh set.

To gate manually, compare the written --out file against the baseline on newly generated cases, then promote if it wins:

moda prompts ab --baseline-key=support.triage \
  --candidate=prompts/support/triage-candidate.prompt.md --auto-generate

moda prompts promote support.triage --label=prod --version=<new_version_id>

Pass the candidate as the --out file path, not as the prompt key. The candidate exists only as an unlabeled registry version; --candidate-key=support.triage resolves to the baseline content on disk and compares the prompt against itself.

Interactive comparisons: the Playground

For hands-on iteration, open Dashboard → Playground (in the Evaluate group), or click Open in Playground on a prompt’s detail page. The Playground compares the prompt’s production content against a proposed edit over scenario or eval-set datasets, and lets you save an edit as a proposal for review. Everything connects back to the prompt’s Optimize tab: proposals and AI-generated candidates queue there for review, comparison runs appear in its run history, and merged candidates become unlabeled versions that you promote from the dashboard or with moda prompts promote.

Next steps

Prompt workflow — sync and promote, including the lockfile the experiment arms read.
Prompt attribution — link production calls to versions so comparisons reflect real usage.
Evaluate — Replays and the Playground in the dashboard.
CLI reference — output modes and exit codes for moda prompts commands.

Get started

Concepts

Dashboard

Prompt management

Harness

Administration

Prompt experiments

Prerequisites

Compare two prompts: moda prompts ab

Win criteria

Choosing the replay set

Flags

Generate a revised candidate: moda prompts propose

Propose flags

Gating and promoting a candidate

Interactive comparisons: the Playground

Next steps

​Prerequisites

​Compare two prompts: moda prompts ab

​Win criteria

​Choosing the replay set

​Flags

​Generate a revised candidate: moda prompts propose

​Propose flags

​Gating and promoting a candidate

​Interactive comparisons: the Playground

​Next steps

Prerequisites

Compare two prompts: moda prompts ab

Win criteria

Choosing the replay set

Flags

Generate a revised candidate: moda prompts propose

Propose flags

Gating and promoting a candidate

Interactive comparisons: the Playground

Next steps