Skip to content

ai(observability): PostHog holds no token counts for any ORB AI call, though $498 of spend lands correctly #10207

Description

@JSONbored

Part of epic #8286 (Phase 3 — AI observability). ORB-side counterpart to #10198.

Problem

PostHog holds no token data at all for the ORB's AI calls. Verified on project 527730:

  • $ai_input_tokens and $ai_output_tokens do not appear in the property list for $ai_generation at all (read-data-schema, event_properties).
  • posthog.ai_events.input_tokens / output_tokens / total_tokens are NULL for every model over the last 2 days — including claude-sonnet-5, which accounts for 2,321 calls and $498.43 of real spend in the same window.

Cost lands correctly ($ai_total_cost_usd is populated), so the event itself is arriving and being parsed — it is specifically the token fields that carry nothing.

Why this is not simply "the provider reports no tokens"

capturePostHogAiGeneration (src/selfhost/posthog.ts) always sets both properties:

$ai_input_tokens: Number.isFinite(event.inputTokens) ? event.inputTokens : 0,
$ai_output_tokens: Number.isFinite(event.outputTokens) ? event.outputTokens : 0,

So a value is always sent — 0 when the provider reported nothing. Two candidate mechanisms, not yet distinguished:

  1. usage.inputTokens / usage.outputTokens are genuinely undefined for the CLI providers, so 0 is sent for every call and PostHog treats 0 as no-data. extractCliUsage/mergeUsage can populate them — recordCliUsageMetrics guards its Prometheus counters on usage.inputTokens !== undefined — so whether they are populated in practice for claude-code needs checking against real CLI stdout.
  2. The values are populated but dropped somewhere between capture and ingestion.

Worth checking (1) first: if the ORB is sending 0 for every call, that is the same failure shape as #10198 on the miner side — a hardcoded/degenerate 0 where a real figure exists — and the fix is the same in spirit.

Impact

PostHog's LLM cost views break down spend by token usage. With no tokens, the ORB shows dollar cost but no tokens/call, no input-vs-output ratio, and no cost-per-token — and $ai_input_cost_usd/$ai_output_cost_usd, which PostHog derives from tokens, cannot be computed.

This also makes the "AI spend by model" dashboard panel unable to carry token columns; they were removed from it rather than left reading null.

Deliverables

  • Determine which of the two mechanisms applies — capture a real claude-code and ollama call and inspect the AiUsage actually handed to capturePostHogAiGeneration.
  • If the providers do report tokens, ensure they reach the event.
  • If a provider genuinely reports none, OMIT the property rather than sending 0 — a fabricated 0 is indistinguishable from a real 0 in an aggregate, which is the same reasoning applied in ai(observability): the miner reports 0 input/output tokens because both drivers sum the split away #10198.
  • Add the token columns back to the "AI spend by model" panel on the AI observability dashboard once real values land.

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions