fix(codex): ignore duplicate token_count totals

[KosBeg]

Summary

Codex session logs can emit duplicate token_count events where the cumulative total_token_usage does not advance. The loader previously trusted last_token_usage unconditionally, which causes duplicated deltas and inflates all token totals (often close to x2). This change skips last_token_usage when totals are unchanged, and only derives deltas from totals when last is missing.

Why this happens

Codex logs contain both:

• info.last_token_usage: per-event delta
• info.total_token_usage: cumulative totals

In practice, some sessions emit the same token_count payload twice without advancing the cumulative totals. When we sum last_token_usage for every event, those duplicates are counted again even though total_token_usage shows no progress.

Example from real logs (simplified)

Event A (first emission):

• last_token_usage: input=1000, cached=250, output=200, total=1200
• total_token_usage: input=1000, cached=250, output=200, total=1200

Event B (duplicate emission, totals unchanged):

• last_token_usage: input=1000, cached=250, output=200, total=1200
• total_token_usage: input=1000, cached=250, output=200, total=1200

Old behavior:

• Adds Event A delta + Event B delta = 2400 total tokens (double counted)

New behavior:

• Detects that totals did not change for Event B
• Skips Event B and keeps 1200 total tokens (correct)

How the fix works

• Track the previous total_token_usage per session
• If the current total_token_usage equals the previous total, treat the
  event as a duplicate and ignore last_token_usage
• If last_token_usage is missing but totals exist, derive a delta from the
  totals (existing behavior)

What changed

• Added a small helper to compare totals for equality
• Updated the event processing logic to skip duplicated totals
• Added an in-source test to reproduce the duplicate pattern

Summary by CodeRabbit

• Bug Fixes
  • Improved detection and handling of duplicate token usage events to prevent redundant processing and ensure accurate token count tracking.

_{✏️ Tip: You can customize this high-level summary in your review settings.}

[coderabbitai]

📝 Walkthrough

Walkthrough

Modified the data loader to detect and handle duplicate token usage events. Introduces a new isSameRawUsage helper function to compare RawUsage objects for equality. Refactors per-event raw usage selection logic to skip processing when total usage remains unchanged, and adds unit tests validating this deduplication behavior.

Changes

Cohort / File(s)

Change Summary

Duplicate detection logic apps/codex/src/data-loader.ts

Added isSameRawUsage helper to compare RawUsage objects. Modified loadTokenUsageEvents to detect duplicate totals and conditionally select raw usage (prioritizing lastUsage when not a duplicate, falling back to computed delta from totalUsage). Reworked conditional logic to skip zero deltas and handle null cases. Added unit tests verifying duplicate token_count events with unchanged totals produce single event output.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Poem

🐰 A rabbit hops through tokens with care,
Spotting twins where duplicates dare!
Same totals caught, no delta needed—
One event clean, the noise defeated! 🌟
Data flows pure, through dedup's grace.

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title 'fix(codex): ignore duplicate token_count totals' directly and concisely describes the main change—detecting and ignoring duplicate token_count events when totals are unchanged.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing touches

• 📝 Generate docstrings

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[KosBeg]

This PR fixes an issue where ccusage can double-count Codex token_count events when the JSONL stream contains duplicated token_count messages with unchanged total_token_usage.

I hit this in the wild with a single Codex session (no other sessions/logs). That one session consumed the entire ~5h Plus usage window, and Codex CLI reported the following totals at the end:

• input: 729,985
• output: 261,505 (reasoning: 200,512)
• cached: 45,851,008
• total (input + output): 991,490
• total tokens incl. cache: 991,490 + 45,851,008 = 46,842,498

Before vs after (ccusage)

Metric	Before patch (duplicated)	After patch	Codex CLI
Input	1,459,679	729,985	729,985
Output	515,926	261,505	261,505
Reasoning	394,432	200,512	200,512
Cache read	91,452,800	45,851,008	45,851,008
Total tokens	93,428,405	46,842,498	46,842,498
Cost (USD)	$25.78	$12.96	—

As the screenshots show, before the patch ccusage ended up at ~2× across input/output/reasoning/cache (and cost). After the patch, the numbers match 1:1 with Codex CLI, including cache read and total tokens.

---

Screenshots

1. Codex CLI (limit hit + token usage):

2. ccusage BEFORE patch (~2×):

3. ccusage AFTER patch (matches Codex CLI):