Skip to main content

Cost attribution contracts and delivery lanes

Decision summary

Production still writes cost_usage_event metric rows. The old-ledger shutdown is done (#11640): its dispatch/consumer no longer publishes fresh days. Until the SQL ledger publishes, the published scoreboard is frozen at Nov 12; report progress from shadow builds, not as fresh published results. This PR changes no emitter, app import, database, flag or deployment. It defines the receipt, coverage, basis and ownership contracts for later work. SQL in docs/cost/contracts/ is a proposal, not a migration. A migration requires explicit user approval and a successful dev dry run; only one may be in flight at a time. Do not copy these files into migrations without approval. Lane E first cuts over using the rows we have today. Only after several days of matching both formats does it switch to cost_call, deleting the old metric-matching code in the same stack. Five B lanes may write code in parallel; their deployments are staggered. Dates follow validation, not agent count.

Rung ladder and current work

  1. Owner: ≥95% of bill $ have an owner+class (Nov 12: 43%).
  2. Tenant: ≥90% of serving $ have a tenant, which gives average cost per message per tenant.
  3. Run: ≥90% of tenant $ have run+stage (AI was 45% on Nov 14).
  4. Message: every run records which messages it processed; per-call weights only where they change a decision.
Lanes only take work for the current rung. The current rungs are 1–2: shut down the old ledger (done, #11640) and switch to the SQL ledger; roll out keys (credential_ref 0% today); owner registry plus bill gaps; split Modal and ClickHouse by tenant. The B lanes’ first job is tenant and key on every receipt. Per-message (user re-plan): a per-message v0 query ships now from the rows we have. v1 run membership in moda-problems (C2, moda.cost_run_membership) is unblocked and no longer waits on the run rung. credential_ref and run IDs in other apps are off the per-message critical path. Park the six coverage PRs’ call-ID rework (#11314, #11318, #11324, #11325, #11328, #11331), #11516, #11521, #11528, #11536 and lane D’s per-run/per-conversation bases. Preserve approved work; do not rewrite it while it is parked.

1. Goals, accounting rules and outputs

One AI call becomes one receipt with its usage, scope, timing and charge. A scope is set at the entry point, not guessed downstream. Close each provider/day to its bill, to the nanodollar: every bill dollar appears once. Keep charge credits, named idle costs and an explicit unexplained line; unexplained is never spread. Publication is the commit point: unpublished or failed builds are never read. Outputs stay cost_ledger_daily, cost_interaction_daily and cost_ledger_publications. They provide provider bill closure, shared-cost allocation with residuals, interaction costs, tenant margin and a scoreboard. Ledger kinds remain exact, measured, allocated, internal, unexplained. allocated_estimate describes an allocation method, not an exact receipt. Marginal cost is conversation-level AI cost assigned by coverage, plus serving compute allocated to that conversation. Fully loaded adds tenant work and serving idle in proportion to marginal cost, including fixed serving costs: storage and the serving share of Vercel/Cloudflare. Exclude R&D, CI and retired resources from fully loaded serving cost; keep them internal and visible. Zero marginal cost cannot be a divisor: retain the unallocated pool explicitly until an approved fallback exists. Incomplete coverage remains run-level rather than artificially inflating covered messages’ costs. No runtime integration, production benchmark, ledger rebuild, backfill or schema execution is part of this PR. A new coordinator or other mechanism requires a written reason, not an agent’s preference.

2. Inputs and flow

  • Raw usage: moda.cost_usage_event is unchanged as the current source of truth and Lane E’s first input. Existing sibling suppression, exact-charge selection and fine-grain pricing remain until receipt parity is proven.
  • Receipts: future moda.cost_call has one latest row per billable call. It is written at eventsFromLlmUsage in packages/pipeline-runtime/src/cost.ts and the Python twin in packages/moda-pipeline-runtime-python/src/moda_pipeline_runtime/cost.py. Existing call sites keep their API; B lanes supply attribution and coverage.
  • Coverage: new moda.cost_run_membership, one row per run per day with the complete unit IDs it read; calls join on run_id. No call_id in v1. The legacy moda.cost_usage_coverage is left as it is (no ALTER/DROP). Do not create cost_call_coverage. See C2 below.
  • Bills: provider_cost_fact supplies vendor/account/SKU/day amounts and credits. C preserves restatements, collects missing AWS/Bedrock/Anthropic sources and normalizes cf to cloudflare. #11550’s per-key OpenRouter split is checked against the SQL ledger, not blindly re-landed. An estimate is not an invoice; fixed Vercel cost is booked fixed. Source fingerprints include bill corrections and fetched data.
  • Prices: C owns packages/database/src/cost/price-book.ts, including cache and reasoning rates. Prices are versioned, applied at model/provider/key fine grain, and never override an available provider charge.
  • Owner registry: C supplies infra/cost-owner-registry.yaml and loads the proposed table. Resource types cover app, credential, Modal app, Cloudflare Worker and ClickHouse service. Owners have serving, rnd, ci, retired classes and non-overlapping effective windows. Missing ownership is visible; it is never silently treated as serving.
  • Shared bases: cost_basis_daily stores ClickHouse CPU-seconds, Modal container/busy-seconds, Cloudflare usage and explicitly versioned estimates. Quantity, unit and basis version travel with the attribution grain. Idle is resource='idle', tenant_id='', method='idle'. Measured usage comes first: tokens, busy seconds or exact cost. Message count is only an explicitly named fallback, with method='allocated_estimate' and a basis version. Nov 13 message shares (tenants 6d9f/2290/4bd5) were 81.5/12.1/6.4%; Modal embeddings hours were 91.3/4.3/4.4%; reported AI-dollar shares were 8.7/86.5/5.1%. The last series is a reported diagnostic, not normalized here. These differ too much for message count to be a primary cost basis. Untagged Modal time (about 200 hours that day) must remain separately labeled when estimated.
  • Reports: F reads only published ledger outputs, not another independent raw-usage pricing or allocation path. Existing approved report PRs are parked or adopted as listed in pr-lane-map.md.
Approved future volume changes retain their meaning: hourly Cloudflare queue_operations/bytes_written, rollups of streams below $1/day, no always-zero embeddings output-token row, duration rollups preserving totals/counts/latency distribution and tenant work bases. Hourly grains retain tenant, source, stage, metric, provider/account/resource and operation/class; preserve conversation/run dimensions where consumed, or a narrow identity mapping. No merge is allowed until reader/inventory/no-row tests show the chosen grain preserves those uses.

3. Contract C1: one receipt per billable call

Types and validators live only in packages/cost-contract, not yet imported by apps. TS and Python share fixtures/contracts.json and a cross-language test.

Identity and corrections

call_id is lowercase SHA-256 hex of the UTF-8 canonical JSON array:
After the version tag, these are the canonical inputs of makeCostIdempotencyKey, minus revision, metric, usage_class, interval_id and seq (including their aliases). Every array element is a string. Serialize with no insignificant whitespace: TS uses JSON.stringify; Python uses json.dumps(..., ensure_ascii=False, separators=(',', ':')). Both hash the UTF-8 bytes, with JSON escaping for quotes, backslashes and control characters and no Unicode normalization or timestamp defaults. The literal cost_call_v1 tag is independent of the correction revision. tenant_id ?? tenant ?? '' preserves the old alias/empty-string semantics. Other missing identity fields are empty. A missing source or source event ID returns an empty call ID; it must be counted and rejected, not randomly filled. JSON preserves field boundaries: s/e1/t and s/e/1t produce different IDs. Source event IDs must still be stable and unique per actual call. The legacy metric-level makeCostIdempotencyKey stays unchanged; its delimiter-free encoding is not used for call IDs. Another encoding requires a new version tag and an explicitly approved contract change, not a lane edit. Every metric row of the same call shares this identity. A billable retry is a new call and must have a distinct stable source event/attempt identity; a write retry is the same call and revision. revision is a positive safe integer and the SQL version column. Calls use PARTITION BY toYYYYMM(day) and ORDER BY (day, tenant_id, call_id), with a 730-day TTL like raw usage. Deduplication works only within the sorting key and partition. Day and tenant are immutable for the life of a call; daily builds filter day rather than scanning all history. Day is derived from ended_at in UTC. A time correction crossing midnight requires a reversal on the old day plus a new call identity, like other identity changes, not moving a revision to a new key. Equal key/version retries have identical data. Readers select the latest revision explicitly; background merges are not a correctness mechanism. The shared whitespace rule is explicit: only ASCII space, tab, CR and LF count as blank; they are not trimmed from otherwise present IDs. BOM (U+FEFF) and unit separator (U+001F) are nonblank. Both languages reject malformed Unicode (lone surrogates) with validation errors, never a validator crash. Python also catches UnicodeEncodeError. Shared fixtures enforce the same hashes/validation results, including BOM, U+001F and lone surrogates. Coverage keyed on idempotency_key is wrong: that key identifies one metric row (input tokens, output tokens, charge, etc.), not the billable call. Joining it to coverage drops or duplicates parts of the same call. Do not hash that metric key to manufacture a call ID.

Charge, usage, timing and required scope

  • Provider-reported provider_cost_usd, including zero or a signed correction, wins: cost_source='exact', cost_usd=provider_cost_usd. Only null means no charge was supplied; then use the price book’s measured_cost_usd and cost_source='measured'. resolveCharge and validation enforce this.
  • Input includes cached tokens; output includes reasoning tokens. Cache read and write are disjoint input subsets; reasoning is an output subset. Never add breakdowns to totals. Prices apply once to the relevant subdivisions, not once to totals and again to cached/reasoning tokens. normalizeUsage in TS/Python converts provider formats before validation: Anthropic and bedrock-claude report uncached input separately, so 10 input
    • 7 cache-read + 3 cache-write becomes 20 inclusive input. OpenRouter and openai-style already include cache: 100 prompt with 40 cached stays 100. Reasoning remains inside output. Shared fixtures exercise all four formats.
  • started_at and ended_at are UTC DateTime64(3), represented on the wire as ISO UTC strings with exactly three decimal places. End must be at least start. day is the immutable UTC end date; requests is one. Counts are nonnegative safe integers shared by TS/Python, with UInt64 SQL storage.
  • conversation requires tenant_id, conversation_id and run_id.
  • tenant requires tenant_id, run_kind, run_id.
  • company requires initiator_kind, initiator_id, and empty tenant_id. The receipt level is company, while ledger classification is internal.
  • LLM calls require credential_ref (a name, never the key secret). The explicit resource_kind='llm' also catches newly added vendors. Known LLM provider names require it even if mislabeled other.
  • Validation rejects every missing required field. Missing IDs stay empty and appear in missing_ids, which Lane A must count. This package never inserts random IDs, placeholder tenants, inferred conversations or UUID defaults.

Sub-nanodollar money

All unit prices (USD per one token, not per million) and call costs are Decimal(38,18), passed as decimal strings. Do not pass JavaScript floats. Price-book computations use decimal arithmetic; no call-level nanos conversion is allowed. Round to integer nanos once at the ledger slice, after summing its high-precision amounts. Apply truncation plus an explicit residual and deterministic largest remainder when distributing a slice; allocations plus their residual must equal the bill slice exactly. Persist the pricing/basis/method version used. No sub-nanodollar charge is lost merely because it is individually smaller than one nano.

4. Contract C2: run membership in stable message units

moda.cost_run_membership holds one row per run per charged day (per extraction family, below): the complete list of message UNIT IDs the run actually read, scoped by day, tenant and conversation. IDs are recorded in full, never sampled or truncated. Calls join to it on (day, tenant_id, conversation_id, run_id); per-call membership (call_id) is not part of v1. Source rows the run read that name no unit yet (mapped-source rows, units not materialized) go in unresolved_keys: recorded explicitly, never guessed. Never join metric-level idempotency keys. No message-index ranges: apps/moda-conversation/src/message_units.py:932–944 rewrites indexes, and worker.py:4408–4416 executes rewrites before inserting late batches. Unit IDs survive reordering. Record extraction_version and, for mapped IDs, generation_digest: mapped IDs are generation-scoped. extraction_version is one of the four values production writes to moda.conversation_message_units, grouped in families: Any other value is rejected; a new extraction version needs a contract change. Families cover different message sources, so their unit sets are disjoint and one run can read several: membership is the union of one row per family. In prod on Nov 15 (5% sample of run conversations, 322 conversations), 311 had units in more than one of v2/source_v1/mapped_v1 and no unit_id appeared under two of them; only 105 unit IDs appeared under both v1 and v2 (3 conversations). Within a family, for each (run_id, day) scoped by tenant/conversation, the close uses only the highest-ranked context (v2 supersedes v1); older contexts are ignored, never summed. A unit_id under two families fails closed. Within one version, ties choose the greatest generation_digest by unsigned UTF-8 byte order, not arrival time or revision. Deduplicate revisions within each full SQL key first, then latestCoverage selects the winning context; validateCoverage checks the selected rows. Conflicting equal-key revisions fail closed. Revision does not make an older extraction current. Recorded cost shares remain tied to the original charged membership snapshot: units absent from the latest version retain those shares at conversation level, counted as unresolved, even if they still exist elsewhere. Intersect the latest context’s unit IDs with active units before coverageAllocation; never redistribute the absent shares or add costs from older contexts. Split evenly by default (weight_basis='equal'): one share per unit and per unresolved key; unresolved keys’ shares stay at conversation level and are counted. Proportional token_counts (tokens) and explicit decimal weights are later refinements only where they change a decision, and require no unresolved keys. Fractions are in [0,1] and sum to at most one per run. Never duplicate membership across contexts. Explicit weights are never normalized upward; the remainder stays run-level. Empty membership keeps all cost run-level. If a recorded unit is later merged away or deleted, retain its original share at conversation level and count it as unresolved; never drop it or redistribute it to surviving units. coverageAllocation and shared contract tests enforce this conservation rule. Membership is an observed snapshot, not today’s set of active units. validateCoverage checks membership/context, fractions, one row per run/day/family and no unit under two families; coverageAllocation takes a run’s selected rows together. The table has a day column, monthly partitioning, a sorting key of (day, tenant_id, conversation_id, run_id, extraction_version, generation_digest) and a 730-day TTL. All key fields are immutable. Day is the UTC end day of charged calls; a run spanning days repeats membership per charged day. Daily readers filter day; latest revisions are deduplicated within the full key. Volume (prod, Nov 15, read-only): about 13k moda-problems runs a day (reader 5.0k, task 5.2k, task review 2.5k, Dream about 0.2k), so about 13k rows a day (a little more where runs span families). The legacy prod table moda.cost_usage_coverage is unused (0 rows) and left in place. Creating the new table is a separate migration: dev dry run and user approval first. No backfill.

5. Contracts C3–C6: bases, registry, close and reports

CostBasisDaily grain is day × provider × resource × tenant × conversation × run × app × stage × unit × basis_version × method, versioned by revision. Units are never mixed. Preserve measured bases separately from message estimates. Registry joins use effective windows at call/bill time; overlapping owners fail closed. The SQL files describe both tables without deploying them. E uses set-based SQL and staging keyed by deterministic build/step IDs. The Worker/DO sequences work, not giant in-memory aggregation. Use deterministic keys/insert tokens so an interrupted build resumes its failed step. Pin a source watermark and checksum fence, including bills, prices, registry, coverage and bases; re-verify before publishing and fail closed on change. Write the publication manifest last. Retain the previous publication on any failure. Close day D once by 08:00 UTC on D+1; rebuild only when its input fingerprint changes. Hourly rebuilding costs approximately 9 CPU-h/day, over the 2% budget. Vendor bills can arrive late or be restated: include fetched bill revisions and historical rule/price/registry changes in invalidation, not only raw usage or a two-day timer. Finality follows the vendor correction window; changes after freeze need explicit restatement, never silent loss. Do not add periodic no-change rebuilds. No new 1 GiB profile is approved here; that separate test/design decision remains on hold.

6. Exclusive lanes and existing PR ownership

infra/cost-lanes.json is the executable path map. Specific exclusions separate shared Problems directories and Modal runtime files; the globs, not app names alone, decide ownership. Unowned paths require architect assignment, not a wildcard exception in a lane PR. C owns collector and database manual-bills.ts; E owns database rate.ts and financial-source.ts. Database schema/migrations/** and the repository’s actual src/migrations/** are architect-owned, not lane-specific DDL exceptions. These B groups are ordered by the observed spend priorities: PR tracker, task review, reader, replay, dream/embeddings. Five lanes write in parallel, not five simultaneous deploys. B2/B3/B5 coordinate shared Problems helper changes through their owner; they do not edit each other’s paths. Every existing cost PR has exactly one owner in pr-lane-map.md. For a PR spanning lanes, choose the lane with the largest additions+deletions in owned files; note every other lane/shared path and split before applying strict lane markers. Only the owning lane pushes to that PR. No other session pushes, including Replicas sessions that push review fixes. Existing sessions are not automatically authorized by a historical footer. The architect assigns one active session within the owner lane; stop other sessions before resuming work. No lane rewrites work already in an approved PR. Adopt it and coordinate its merge. The six coverage PRs’ call-ID rework is parked, preserving approved work. Close #11512 and #11545. #11532 was reverted by #11636: do not re-land it. #11523, #11509 and #11550 belong to E for a check against the SQL ledger. #11516/#11521/#11528/#11536 are parked rather than duplicated. No other PR is closed/edited by this contracts change. Historical cross-lane PRs must be split/coordinated before new marker enforcement; they are not blanket exemptions for future edits. CI recognizes cost-lane:<X> labels and cost/<lane>/... branches. Conflicting, unknown or malformed markers fail. Marked PRs may touch only their lane’s paths; renames check both old/new names and overlapping ownership fails. Unmarked PRs fail only when they touch unmarkedCostPaths, the explicit short list in infra/cost-lanes.json: cost emitters/wrappers, database cost code, the whole dedicated cost-collector and Modal-billing apps, owner/key registries, cost emission scripts, cost/deploy workflows, Python cost-runtime/container-interval instrumentation, contracts/docs, filename-matched cost migrations, admin cost pages and the lane controls themselves. Whole app ownership otherwise applies only to lane-marked PRs: ordinary Problems or runtime changes need no cost marker. Root package.json and bun.lock remain outside this short list (19 merged PRs touched them in 14 days). Lane-marked PRs may touch them only as architect. Contract tests still run. Ownership is checked using the base commit’s policy and checker before executing head code, so changing the PR’s map cannot authorize its own edits. The first policy-creating PR necessarily bootstraps from its head; that initial policy needs explicit review. The workflow resolves queue origins and checks source diffs and queue paths with the base checker first. The head runner adds required-body and Graphite queue-origin checks; see lane operating rules. CI checks paths, not human identity: the single-session pushing rule also needs operator enforcement. Making this check required in merge rules needs a repository admin.

7. Deployment radius and canaries

Path-filter audit: a change to TS runtime src/cost.ts matches 13 deploy workflows; the Python twin matches 18. There are 25 unique filtered workflows across the two. These are workflow counts, not a count of distinct apps/functions: the Problems workflow deploys both Problems and Dream, for example. An additional frontend workflow runs unfiltered on every main push. TS: analytics-scheduler, ingest, contract-compile, contract-validation, embeddings, github, harness-analyze, replay-runner, sentry-tail, thread-segments, tool-classification, topic-assignment, world-facts (deploy-*.yml names). Python: analytics-container, clustering, compliance-agent, conversation-drainer, conversation-worker, custom-signals-agent, design-registry, frustration-agent, laziness-agent, contract-compile, contract-validation, embeddings, problems, replay-runner, tool-classification, topic-assignment, user-refresh, replay. Lane A ships behind a default-off per-app flag. Package deployment may fan out as above, but enable receipt emission for one app at a time. Before each enable, record a baseline and rollback literal. Observe at least 30 minutes for insert loss/retries, inference/error rate, hook latency and volume. Any error rise means revert; keep the existing p99 >500 ms / interval volume >50k/hour rollback thresholds where that instrumentation is applicable. Confirmed inserts require wait_for_async_insert=1, bounded retries and surfaced shutdown loss. Run B deployments one at a time with the same canary rule. Do not re-enable Problems interval recording as a side effect of receipt adoption. The standalone contract module has no workspace manifest or app import and matches no path-filtered app deploy. But zero deploy workflows on merge cannot be promised: deploy-frontend.yml deliberately deploys main’s latest tip without path filtering, including docs-only commits. The user accepts this existing exception: docs/contracts-only changes may harmlessly redeploy the unchanged frontend. This PR does not change that policy. Lane deployments still require the validation and canary gates above.

8. Validation-driven plan, rollback and deletion

  1. Contracts first: review these types, fixtures, SQL and lane assignments. Land no database migration with them. Each future schema change needs the user approval + dev dry run rule, separately, one in flight at a time.
  2. E first cutover: old-ledger shutdown is done (#11640). #11633 is lane E’s next job after #11634 merges, not parked; rung 1 depends on its SQL-ledger coordinator and publication work. Do not re-land #11532 (reverted by #11636); close #11512. Shadow Nov 11–12 using current narrow rows and compare against those old published days. Publish only once bill totals are equal and attribution differences are explained. Publications are provisional during the seven-day acceptance window. buildLedger is kept only as a frozen test comparison, never a runtime fallback. Until this publishes, progress comes from shadows; the published scoreboard stays frozen at Nov 12. Target ≤2% ClickHouse CPU; build D once at D+1 and rebuild only on input change, not hourly. The old-engine deletion PR lands within days in the same stack as the stop PR, independently of later receipt adoption. Keep only the frozen test comparison. Staging-table drops need explicit user approval and a dev dry run.
  3. A and C/D: approved schemas precede writers. A canaries one app and then dual-writes confirmed receipts; compare receipt dollars/usage/identity to raw rows for at least three full days, including corrections/retries. C/D adopt approved bills/bases PRs; changes that need migrations wait for approval.
  4. B1–B5: code may proceed in parallel against the contract. Enable in spend order, staggered, with stage-level credential/scope/coverage scoreboards.
  5. E receipt input: only after several days of matching formats, switch to receipts + run membership + tenant bases + registry. Repeat shadows/row diffs and seven consecutive days of acceptance: exact bill totals, no lost calls, correct residuals, publication by 08:00, measured cost within budget. Delete obsolete format-matching code in the same receipt-adoption stack. This does not defer the old-engine deletion above. The seven-day gate is validation, not a promise that five agents finish in a week.
  6. F: unpark reports only when publication/input contracts are accepted. Reconcile one month of tenant margins to bills before report cutover.
There is no rollback to the old ledger path. Rollback means the last publication stays; failed or mismatched new builds never replace it. Preserve raw facts and recorded membership. Legacy rating (rating.ts -> cost_rated_daily) keeps running until its Cloudflare-estimate dependency is replaced; deleting the old ledger does not authorize stopping it. Every PR states its lane/step, scoreboard number/proof query and deleted code. Use Graphite (gt modify -cam, then gt submit --no-stack --no-edit --publish --no-interactive) only when authorized; require exact-head Greptile 5/5, Replicas approval and green CI before merge.

9. Open gates and risks

  • User approval and dev dry run for each proposed schema, especially the new cost_run_membership table; SQL syntax/Cloud shared-engine adaptation is not claimed production-tested by this contracts PR.
  • Stable producer source/attempt identities and legacy-to-call mapping must be demonstrated. Missing IDs are counted/rejected, never randomly repaired.
  • Equal-revision conflicting payloads, ownership-window overlap, changed source fences, invalid coverage and unknown prices must prevent publication.
  • Decimal precision must survive collector/JSON/driver boundaries; ledger-slice rounding and largest remainder need SQL integration tests in E’s lane.
  • Actual close CPU/bytes/memory and vendor correction windows need validation; the held 1 GiB test is not implicitly resumed.
  • PR footers identify associated sessions, not a verified last GitHub pusher. Where no session is visible, stop/assign through the PR author before edits.