Cost attribution contracts and delivery lanes
Decision summary
Production still writescost_usage_event metric rows. The old-ledger shutdown
is done (#11640): its dispatch/consumer no longer publishes fresh days. Until
the SQL ledger publishes, the published scoreboard is frozen at Nov 12; report
progress from shadow builds, not as fresh published results.
This PR changes no emitter, app import, database, flag or deployment.
It defines the receipt, coverage, basis and ownership contracts for later work.
SQL in docs/cost/contracts/ is a proposal, not a migration. A migration
requires explicit user approval and a successful dev dry run; only one may be
in flight at a time. Do not copy these files into migrations without approval.
Lane E first cuts over using the rows we have today. Only after several days
of matching both formats does it switch to cost_call, deleting the old
metric-matching code in the same stack. Five B lanes may write code in parallel;
their deployments are staggered. Dates follow validation, not agent count.
Rung ladder and current work
- Owner: ≥95% of bill $ have an owner+class (Nov 12: 43%).
- Tenant: ≥90% of serving $ have a tenant, which gives average cost per message per tenant.
- Run: ≥90% of tenant $ have run+stage (AI was 45% on Nov 14).
- Message: every run records which messages it processed; per-call weights only where they change a decision.
credential_ref 0% today); owner registry plus bill gaps; split Modal
and ClickHouse by tenant. The B lanes’ first job is tenant and key on
every receipt.
Per-message (user re-plan): a per-message v0 query ships now from the rows
we have. v1 run membership in moda-problems (C2, moda.cost_run_membership)
is unblocked and no longer waits on the run rung. credential_ref and run IDs
in other apps are off the per-message critical path.
Park the six coverage PRs’ call-ID rework (#11314, #11318, #11324, #11325,
#11328, #11331), #11516, #11521, #11528, #11536 and lane D’s per-run/per-conversation
bases. Preserve approved work; do not rewrite it while it is parked.
1. Goals, accounting rules and outputs
One AI call becomes one receipt with its usage, scope, timing and charge. A scope is set at the entry point, not guessed downstream. Close each provider/day to its bill, to the nanodollar: every bill dollar appears once. Keep charge credits, named idle costs and an explicit unexplained line; unexplained is never spread. Publication is the commit point: unpublished or failed builds are never read. Outputs staycost_ledger_daily, cost_interaction_daily and
cost_ledger_publications. They provide provider bill closure, shared-cost
allocation with residuals, interaction costs, tenant margin and a scoreboard.
Ledger kinds remain exact, measured, allocated, internal, unexplained.
allocated_estimate describes an allocation method, not an exact receipt.
Marginal cost is conversation-level AI cost assigned by coverage, plus serving
compute allocated to that conversation. Fully loaded adds tenant work and
serving idle in proportion to marginal cost, including fixed serving costs:
storage and the serving share of Vercel/Cloudflare. Exclude R&D, CI and retired
resources from fully loaded serving cost; keep them internal and visible.
Zero marginal cost cannot be a divisor: retain the unallocated pool
explicitly until an approved fallback exists. Incomplete coverage remains
run-level rather than artificially inflating covered messages’ costs.
No runtime integration, production benchmark, ledger rebuild, backfill or schema
execution is part of this PR. A new coordinator or other mechanism requires a
written reason, not an agent’s preference.
2. Inputs and flow
- Raw usage:
moda.cost_usage_eventis unchanged as the current source of truth and Lane E’s first input. Existing sibling suppression, exact-charge selection and fine-grain pricing remain until receipt parity is proven. - Receipts: future
moda.cost_callhas one latest row per billable call. It is written ateventsFromLlmUsageinpackages/pipeline-runtime/src/cost.tsand the Python twin inpackages/moda-pipeline-runtime-python/src/moda_pipeline_runtime/cost.py. Existing call sites keep their API; B lanes supply attribution and coverage. - Coverage: new
moda.cost_run_membership, one row per run per day with the complete unit IDs it read; calls join on run_id. Nocall_idin v1. The legacymoda.cost_usage_coverageis left as it is (no ALTER/DROP). Do not createcost_call_coverage. See C2 below. - Bills:
provider_cost_factsupplies vendor/account/SKU/day amounts and credits. C preserves restatements, collects missing AWS/Bedrock/Anthropic sources and normalizescftocloudflare. #11550’s per-key OpenRouter split is checked against the SQL ledger, not blindly re-landed. An estimate is not an invoice; fixed Vercel cost is booked fixed. Source fingerprints include bill corrections and fetched data. - Prices: C owns
packages/database/src/cost/price-book.ts, including cache and reasoning rates. Prices are versioned, applied at model/provider/key fine grain, and never override an available provider charge. - Owner registry: C supplies
infra/cost-owner-registry.yamland loads the proposed table. Resource types cover app, credential, Modal app, Cloudflare Worker and ClickHouse service. Owners haveserving,rnd,ci,retiredclasses and non-overlapping effective windows. Missing ownership is visible; it is never silently treated as serving. - Shared bases:
cost_basis_dailystores ClickHouse CPU-seconds, Modal container/busy-seconds, Cloudflare usage and explicitly versioned estimates. Quantity, unit and basis version travel with the attribution grain. Idle isresource='idle',tenant_id='',method='idle'. Measured usage comes first: tokens, busy seconds or exact cost. Message count is only an explicitly named fallback, withmethod='allocated_estimate'and a basis version. Nov 13 message shares (tenants 6d9f/2290/4bd5) were 81.5/12.1/6.4%; Modal embeddings hours were 91.3/4.3/4.4%; reported AI-dollar shares were 8.7/86.5/5.1%. The last series is a reported diagnostic, not normalized here. These differ too much for message count to be a primary cost basis. Untagged Modal time (about 200 hours that day) must remain separately labeled when estimated. - Reports: F reads only published ledger outputs, not another independent
raw-usage pricing or allocation path. Existing approved report PRs are parked
or adopted as listed in
pr-lane-map.md.
queue_operations/bytes_written, rollups of streams below $1/day, no always-zero
embeddings output-token row, duration rollups preserving totals/counts/latency
distribution and tenant work bases. Hourly grains retain tenant, source, stage,
metric, provider/account/resource and operation/class; preserve conversation/run
dimensions where consumed, or a narrow identity mapping. No merge is allowed
until reader/inventory/no-row tests show the chosen grain preserves those uses.
3. Contract C1: one receipt per billable call
Types and validators live only inpackages/cost-contract, not yet imported by
apps. TS and Python share fixtures/contracts.json and a cross-language test.
Identity and corrections
call_id is lowercase SHA-256 hex of the UTF-8 canonical JSON array:
makeCostIdempotencyKey,
minus revision, metric, usage_class, interval_id and seq (including their
aliases). Every array element is a string. Serialize with no insignificant
whitespace: TS uses JSON.stringify; Python uses
json.dumps(..., ensure_ascii=False, separators=(',', ':')). Both hash the UTF-8
bytes, with JSON escaping for quotes, backslashes and control characters and no
Unicode normalization or timestamp defaults. The literal cost_call_v1 tag is
independent of the correction revision.
tenant_id ?? tenant ?? '' preserves the old alias/empty-string semantics.
Other missing identity fields are empty. A missing source or source event ID
returns an empty call ID; it must be counted and rejected, not randomly filled.
JSON preserves field boundaries: s/e1/t and s/e/1t produce different
IDs. Source event IDs must still be stable and unique per actual call. The legacy
metric-level makeCostIdempotencyKey stays unchanged; its delimiter-free encoding
is not used for call IDs. Another encoding requires a new version tag and an
explicitly approved contract change, not a lane edit.
Every metric row of the same call shares this identity. A billable retry is a
new call and must have a distinct stable source event/attempt identity; a write
retry is the same call and revision. revision is a positive safe integer and
the SQL version column. Calls use PARTITION BY toYYYYMM(day) and
ORDER BY (day, tenant_id, call_id), with a 730-day TTL like raw usage.
Deduplication works only within the sorting key and partition. Day and tenant
are immutable for the life of a call; daily builds filter day rather than
scanning all history. Day is derived from ended_at in UTC. A time correction
crossing midnight requires a reversal on the old day plus a new call identity,
like other identity changes, not moving a revision to a new key. Equal
key/version retries have identical data. Readers select the latest revision
explicitly; background merges are not a correctness mechanism.
The shared whitespace rule is explicit: only ASCII space, tab, CR and LF
count as blank; they are not trimmed from otherwise present IDs. BOM (U+FEFF)
and unit separator (U+001F) are nonblank. Both languages reject malformed
Unicode (lone surrogates) with validation errors, never a validator crash.
Python also catches UnicodeEncodeError. Shared fixtures enforce the same
hashes/validation results, including BOM, U+001F and lone surrogates.
Coverage keyed on idempotency_key is wrong: that key identifies one metric
row (input tokens, output tokens, charge, etc.), not the billable call. Joining
it to coverage drops or duplicates parts of the same call. Do not hash that
metric key to manufacture a call ID.
Charge, usage, timing and required scope
- Provider-reported
provider_cost_usd, including zero or a signed correction, wins:cost_source='exact',cost_usd=provider_cost_usd. Only null means no charge was supplied; then use the price book’smeasured_cost_usdandcost_source='measured'.resolveChargeand validation enforce this. - Input includes cached tokens; output includes reasoning tokens. Cache read
and write are disjoint input subsets; reasoning is an output subset. Never
add breakdowns to totals. Prices apply once to the relevant subdivisions,
not once to totals and again to cached/reasoning tokens.
normalizeUsagein TS/Python converts provider formats before validation: Anthropic andbedrock-claudereport uncached input separately, so 10 input- 7 cache-read + 3 cache-write becomes 20 inclusive input. OpenRouter and
openai-stylealready include cache: 100 prompt with 40 cached stays 100. Reasoning remains inside output. Shared fixtures exercise all four formats.
- 7 cache-read + 3 cache-write becomes 20 inclusive input. OpenRouter and
started_atandended_atare UTCDateTime64(3), represented on the wire as ISO UTC strings with exactly three decimal places. End must be at least start.dayis the immutable UTC end date; requests is one. Counts are nonnegative safe integers shared by TS/Python, with UInt64 SQL storage.conversationrequirestenant_id,conversation_idandrun_id.tenantrequirestenant_id,run_kind,run_id.companyrequiresinitiator_kind,initiator_id, and emptytenant_id. The receipt level iscompany, while ledger classification isinternal.- LLM calls require
credential_ref(a name, never the key secret). The explicitresource_kind='llm'also catches newly added vendors. Known LLM provider names require it even if mislabeledother. - Validation rejects every missing required field. Missing IDs stay empty and
appear in
missing_ids, which Lane A must count. This package never inserts random IDs, placeholder tenants, inferred conversations or UUID defaults.
Sub-nanodollar money
All unit prices (USD per one token, not per million) and call costs areDecimal(38,18), passed as decimal strings.
Do not pass JavaScript floats. Price-book computations use decimal arithmetic;
no call-level nanos conversion is allowed. Round to integer nanos once at the
ledger slice, after summing its high-precision amounts. Apply truncation plus
an explicit residual and deterministic largest remainder when distributing a
slice; allocations plus their residual must equal the bill slice exactly.
Persist the pricing/basis/method version used. No sub-nanodollar charge is lost
merely because it is individually smaller than one nano.
4. Contract C2: run membership in stable message units
moda.cost_run_membership holds one row per run per charged day (per
extraction family, below): the complete list of message UNIT IDs the run
actually read, scoped by day, tenant and conversation. IDs are recorded in full, never sampled or truncated. Calls
join to it on (day, tenant_id, conversation_id, run_id); per-call membership
(call_id) is not part of v1. Source rows the run read that name no unit yet
(mapped-source rows, units not materialized) go in unresolved_keys: recorded
explicitly, never guessed. Never join metric-level idempotency keys.
No message-index ranges: apps/moda-conversation/src/message_units.py:932–944
rewrites indexes, and worker.py:4408–4416 executes rewrites before inserting
late batches. Unit IDs survive reordering. Record extraction_version and,
for mapped IDs, generation_digest: mapped IDs are generation-scoped.
extraction_version is one of the four values production writes to
moda.conversation_message_units, grouped in families:
Any other value is rejected; a new extraction version needs a contract change.
Families cover different message sources, so their unit sets are disjoint and
one run can read several: membership is the union of one row per family. In
prod on Nov 15 (5% sample of run conversations, 322 conversations), 311 had
units in more than one of v2/source_v1/mapped_v1 and no unit_id appeared under
two of them; only 105 unit IDs appeared under both v1 and v2 (3 conversations).
Within a family, for each
(run_id, day) scoped by tenant/conversation, the
close uses only the highest-ranked context (v2 supersedes v1); older contexts
are ignored, never summed. A unit_id under two families fails closed. Within
one version, ties choose the greatest generation_digest by unsigned UTF-8 byte
order, not arrival time or revision. Deduplicate revisions within each full SQL key first, then
latestCoverage selects the winning context; validateCoverage checks the
selected rows. Conflicting equal-key revisions fail closed. Revision does not
make an older extraction current.
Recorded cost shares remain tied to the original charged membership snapshot:
units absent from the latest version retain those shares at conversation
level, counted as unresolved, even if they still exist elsewhere. Intersect
the latest context’s unit IDs with active units before coverageAllocation;
never redistribute the absent shares or add costs from older contexts.
Split evenly by default (weight_basis='equal'): one share per unit and per
unresolved key; unresolved keys’ shares stay at conversation level and are
counted. Proportional token_counts (tokens) and explicit decimal weights
are later refinements only where they change a decision, and require no
unresolved keys. Fractions are in [0,1] and sum to at most one per run. Never
duplicate membership across contexts. Explicit weights are never normalized
upward; the remainder stays run-level. Empty membership keeps all cost run-level.
If a recorded unit is later merged away or deleted, retain its original share
at conversation level and count it as unresolved; never drop it or redistribute
it to surviving units. coverageAllocation and shared contract tests enforce
this conservation rule. Membership is an observed snapshot, not today’s set
of active units. validateCoverage checks membership/context, fractions,
one row per run/day/family and no unit under two families; coverageAllocation
takes a run’s selected rows together.
The table has a day column, monthly partitioning, a sorting key of
(day, tenant_id, conversation_id, run_id, extraction_version, generation_digest)
and a 730-day TTL. All key fields are immutable. Day is the UTC end day of
charged calls; a run spanning days repeats membership per charged day.
Daily readers filter day; latest revisions are deduplicated within the full key.
Volume (prod, Nov 15, read-only): about 13k moda-problems runs a day
(reader 5.0k, task 5.2k, task review 2.5k, Dream about 0.2k), so about 13k
rows a day (a little more where runs span families). The legacy prod table
moda.cost_usage_coverage is unused (0 rows) and left in place. Creating the new table is a separate migration:
dev dry run and user approval first. No backfill.
5. Contracts C3–C6: bases, registry, close and reports
CostBasisDaily grain is day × provider × resource × tenant × conversation ×
run × app × stage × unit × basis_version × method, versioned by revision.
Units are never mixed. Preserve measured bases separately from message estimates.
Registry joins use effective windows at call/bill time; overlapping owners
fail closed. The SQL files describe both tables without deploying them.
E uses set-based SQL and staging keyed by deterministic build/step IDs. The
Worker/DO sequences work, not giant in-memory aggregation. Use deterministic
keys/insert tokens so an interrupted build resumes its failed step. Pin a source
watermark and checksum fence, including bills, prices, registry, coverage and
bases; re-verify before publishing and fail closed on change. Write the
publication manifest last. Retain the previous publication on any failure.
Close day D once by 08:00 UTC on D+1; rebuild only when its input fingerprint
changes. Hourly rebuilding costs approximately 9 CPU-h/day, over the 2% budget.
Vendor bills can arrive late or
be restated: include fetched bill revisions and historical rule/price/registry
changes in invalidation, not only raw usage or a two-day timer. Finality follows
the vendor correction window; changes after freeze need explicit restatement,
never silent loss. Do not add periodic no-change rebuilds. No new 1 GiB profile is approved here;
that separate test/design decision remains on hold.
6. Exclusive lanes and existing PR ownership
infra/cost-lanes.json is the executable path map. Specific exclusions separate
shared Problems directories and Modal runtime files; the globs, not app names
alone, decide ownership. Unowned paths require architect assignment, not a
wildcard exception in a lane PR.
C owns collector and database
manual-bills.ts; E owns database rate.ts and
financial-source.ts. Database schema/migrations/** and the repository’s
actual src/migrations/** are architect-owned, not lane-specific DDL exceptions.
These B groups are ordered by the observed spend priorities: PR tracker, task
review, reader, replay, dream/embeddings. Five lanes write in parallel, not
five simultaneous deploys. B2/B3/B5 coordinate shared Problems helper changes
through their owner; they do not edit each other’s paths.
Every existing cost PR has exactly one owner in pr-lane-map.md. For a PR
spanning lanes, choose the lane with the largest additions+deletions in owned
files; note every other lane/shared path and split before applying strict lane
markers. Only the owning lane pushes to that PR. No other session pushes,
including Replicas sessions that push review fixes. Existing sessions are not automatically
authorized by a historical footer. The architect assigns one active session
within the owner lane; stop other sessions before resuming work.
No lane rewrites work already in an approved PR. Adopt it and coordinate its
merge. The six coverage PRs’ call-ID rework is parked, preserving approved work.
Close #11512 and #11545. #11532 was reverted by #11636: do not re-land it.
#11523, #11509 and #11550 belong to E for a check against the SQL ledger.
#11516/#11521/#11528/#11536 are parked rather than duplicated. No other PR is closed/edited by this
contracts change. Historical cross-lane PRs must be split/coordinated before
new marker enforcement; they are not blanket exemptions for future edits.
CI recognizes cost-lane:<X> labels and cost/<lane>/... branches. Conflicting,
unknown or malformed markers fail. Marked PRs may touch only their lane’s paths;
renames check both old/new names and overlapping ownership fails. Unmarked PRs
fail only when they touch unmarkedCostPaths, the explicit short list in
infra/cost-lanes.json: cost emitters/wrappers, database cost code, the whole
dedicated cost-collector and Modal-billing apps, owner/key registries, cost
emission scripts, cost/deploy workflows, Python cost-runtime/container-interval
instrumentation, contracts/docs, filename-matched cost migrations, admin cost
pages and the lane controls themselves. Whole app ownership otherwise applies
only to lane-marked PRs: ordinary
Problems or runtime changes need no cost marker. Root package.json and
bun.lock remain outside this short list (19 merged PRs touched them in 14 days).
Lane-marked PRs may touch them only
as architect. Contract tests still run. Ownership is checked using
the base commit’s policy and checker before executing head code, so changing the PR’s map cannot authorize its
own edits. The first policy-creating PR necessarily bootstraps from its head;
that initial policy needs explicit review. The workflow resolves queue origins
and checks source diffs and queue paths with the base checker first. The head runner adds required-body
and Graphite queue-origin checks; see lane operating rules.
CI checks paths, not human identity:
the single-session pushing rule also needs operator enforcement.
Making this check required in merge rules needs a repository admin.
7. Deployment radius and canaries
Path-filter audit: a change to TS runtimesrc/cost.ts matches 13 deploy
workflows; the Python twin matches 18. There are 25 unique filtered
workflows across the two. These are workflow counts, not a count of distinct
apps/functions: the Problems workflow deploys both Problems and Dream, for
example. An additional frontend workflow runs unfiltered on every main push.
TS: analytics-scheduler, ingest, contract-compile, contract-validation,
embeddings, github, harness-analyze, replay-runner, sentry-tail, thread-segments,
tool-classification, topic-assignment, world-facts (deploy-*.yml names).
Python: analytics-container, clustering, compliance-agent, conversation-drainer,
conversation-worker, custom-signals-agent, design-registry, frustration-agent,
laziness-agent, contract-compile, contract-validation, embeddings, problems,
replay-runner, tool-classification, topic-assignment, user-refresh, replay.
Lane A ships behind a default-off per-app flag. Package deployment may fan
out as above, but enable receipt emission for one app at a time. Before each
enable, record a baseline and rollback literal. Observe at least 30 minutes for
insert loss/retries, inference/error rate, hook latency and volume. Any error
rise means revert; keep the existing p99 >500 ms / interval volume >50k/hour
rollback thresholds where that instrumentation is applicable. Confirmed inserts
require wait_for_async_insert=1, bounded retries and surfaced shutdown loss.
Run B deployments one at a time with the same canary rule. Do not re-enable
Problems interval recording as a side effect of receipt adoption.
The standalone contract module has no workspace manifest or app import and
matches no path-filtered app deploy. But zero deploy workflows on merge
cannot be promised: deploy-frontend.yml deliberately deploys main’s latest
tip without path filtering, including docs-only commits. The user accepts this
existing exception: docs/contracts-only changes may harmlessly redeploy the
unchanged frontend. This PR does not change that policy. Lane deployments
still require the validation and canary gates above.
8. Validation-driven plan, rollback and deletion
- Contracts first: review these types, fixtures, SQL and lane assignments. Land no database migration with them. Each future schema change needs the user approval + dev dry run rule, separately, one in flight at a time.
- E first cutover: old-ledger shutdown is done (#11640).
#11633 is lane E’s next job after #11634 merges, not parked; rung 1
depends on its SQL-ledger coordinator and publication work.
Do not re-land #11532 (reverted by #11636); close #11512. Shadow Nov 11–12 using current
narrow rows and compare against those old published days. Publish only once
bill totals are equal and attribution differences are explained. Publications
are provisional during the seven-day acceptance window.
buildLedgeris kept only as a frozen test comparison, never a runtime fallback. Until this publishes, progress comes from shadows; the published scoreboard stays frozen at Nov 12. Target ≤2% ClickHouse CPU; build D once at D+1 and rebuild only on input change, not hourly. The old-engine deletion PR lands within days in the same stack as the stop PR, independently of later receipt adoption. Keep only the frozen test comparison. Staging-table drops need explicit user approval and a dev dry run. - A and C/D: approved schemas precede writers. A canaries one app and then dual-writes confirmed receipts; compare receipt dollars/usage/identity to raw rows for at least three full days, including corrections/retries. C/D adopt approved bills/bases PRs; changes that need migrations wait for approval.
- B1–B5: code may proceed in parallel against the contract. Enable in spend order, staggered, with stage-level credential/scope/coverage scoreboards.
- E receipt input: only after several days of matching formats, switch to receipts + run membership + tenant bases + registry. Repeat shadows/row diffs and seven consecutive days of acceptance: exact bill totals, no lost calls, correct residuals, publication by 08:00, measured cost within budget. Delete obsolete format-matching code in the same receipt-adoption stack. This does not defer the old-engine deletion above. The seven-day gate is validation, not a promise that five agents finish in a week.
- F: unpark reports only when publication/input contracts are accepted. Reconcile one month of tenant margins to bills before report cutover.
rating.ts -> cost_rated_daily)
keeps running until its Cloudflare-estimate dependency is replaced; deleting
the old ledger does not authorize stopping it. Every PR states its lane/step, scoreboard
number/proof query and deleted code. Use Graphite (gt modify -cam, then
gt submit --no-stack --no-edit --publish --no-interactive) only when authorized;
require exact-head Greptile 5/5, Replicas approval and green CI before merge.
9. Open gates and risks
- User approval and dev dry run for each proposed schema, especially the
new
cost_run_membershiptable; SQL syntax/Cloud shared-engine adaptation is not claimed production-tested by this contracts PR. - Stable producer source/attempt identities and legacy-to-call mapping must be demonstrated. Missing IDs are counted/rejected, never randomly repaired.
- Equal-revision conflicting payloads, ownership-window overlap, changed source fences, invalid coverage and unknown prices must prevent publication.
- Decimal precision must survive collector/JSON/driver boundaries; ledger-slice rounding and largest remainder need SQL integration tests in E’s lane.
- Actual close CPU/bytes/memory and vendor correction windows need validation; the held 1 GiB test is not implicitly resumed.
- PR footers identify associated sessions, not a verified last GitHub pusher. Where no session is visible, stop/assign through the PR author before edits.