Skip to content
BetaBenchAlert v0.1 is in Beta.Numbers are real, but pages and rules can still change.See what changed

Raw data

Logs

Every check our lab has saved: the good ones, the failed ones, and the tokens they cost us. This is the ledger behind every board on the site.

What our lab spent

Tokens the paid apps reported back to us for our own test runs. A token is a small chunk of text, about four letters. Total is new input + cache + output.

Pick a time window

Every count on this page follows the window you pick here. The default is the last 24 hours. 30d is as far back as we serve.

Total tokens · last 5 hours

349,280

62 samples · $0.9681 at published API list prices

Apps reported $0.2501 of their own. 24 of 62 samples report no cost at all — those rows are blank, not free.

10 samples are missing a token count, so the totals above are short by whatever they used.

How to read this

Two dollar columns, and they are not the same. Cost is the figure the app itself sent back with the answer — Claude and Grok send one, Codex sends none. API $ is our own recomputation of the same tokens at the vendor's published API list price, so all three apps sit on one scale. Neither is our bill: we pay flat monthly plans. A blank means not measured, never $0. The same API dollars, rolled up to a whole usage window, are what the quota board prices.

How to read this

New in is uncached prompt tokens. Cache is cache reads plus writes — cheaper, but still usage. Out is everything the model wrote, including think. Total adds those three. Bigger is not better here. It is what our own testing used.

By app

SplitnNew inCacheOutTotalAPI $
Codex24160.3k5,6322,927168.8k$0.4819
Grok Build14160.0k3,7122,991166.7k$0.3397
Claude Code247,03306,68813.7k$0.1465

By check

SplitnNew inCacheOutTotalAPI $
speed50265.0k8,06412.2k285.3k$0.8826
ping1262.3k1,28044664.0k$0.0855

By model

SplitnNew inCacheOutTotalAPI $
Grok 4.6999.1k2,6881,563103.3k$0.2089
Grok 4.5561.0k1,0241,42863.4k$0.1308
GPT-5.6 Luna955.2k077455.9k$0.0120
GPT-5.6 Sol538.3k072939.0k$0.2133
GPT-5.6 Terra537.0k065537.7k$0.0819
GPT-5.5529.8k5,63276936.2k$0.1747
Haiku 4.592,36003,0445,404$0.0176
Sonnet 551,61001,2342,844$0.0156
Fable 551,53301,2052,738$0.0756
Opus 551,53001,2052,735$0.0378

Test runs

5 launches in this window · 13 failed samples. Open a run to see every model it touched and what that cost.

Aug 23, 5:00:00 PMpulsescheduled1m 25s · 13 rec · 0 skip · 6 fail · 38.1k tok
TargetResultWallNew inCacheOutTotalAPI $
pingclaudeok3.48s250075325$0.0006
pingcodexfailedYou've hit your usage limit. Visit https://chatgpt.com/codex/settings/usage to purchase more credits or try again at Aug 24th, 2026 8:30 PM.6.91s0000$0.0000
pinggrokok6.61s9,691256299,976$0.0197
speedclaude/fablerate limit warningserved with a rate limit warning: seven_day_overage_included at 0.76 utilization6.46s3060241547$0.0151
speedcodex/lunafailedYou've hit your usage limit. Visit https://chatgpt.com/codex/settings/usage to purchase more credits or try again at Aug 24th, 2026 8:30 PM.7.09s0000$0.0000
speedcodex/terrafailedYou've hit your usage limit. Visit https://chatgpt.com/codex/settings/usage to purchase more credits or try again at Aug 24th, 2026 8:30 PM.6.42s0000$0.0000
speedcodex/solfailedYou've hit your usage limit. Visit https://chatgpt.com/codex/settings/usage to purchase more credits or try again at Aug 24th, 2026 8:30 PM.5.96s0000$0.0000
speedcodex/gpt-5.5failedYou've hit your usage limit. Visit https://chatgpt.com/codex/settings/usage to purchase more credits or try again at Aug 24th, 2026 8:30 PM.6.33s0000$0.0000
speedgrok/grok-4.6ok7.25s12.3k12829312.7k$0.0264
speedgrok/grok-4.5ok6.25s12.1k25628512.7k$0.0261
speedclaude/haikuok6.62s2720531803$0.0029
speedclaude/sonnetok5.24s3220241563$0.0031
speedclaude/opusok4.80s3060241547$0.0076
Aug 23, 4:10:14 PMpulsecatch-up5.80s · 0 rec · 13 skip · 0 fail · 0 tok
TargetResultWallNew inCacheOutTotalAPI $
pinggrokskipped · already recorded this slot
pingclaudeskipped · already recorded this slot
pingcodexskipped · already recorded this slot
speedclaude/opusskipped · already recorded this slot
speedclaude/fableskipped · already recorded this slot
speedcodex/lunaskipped · already recorded this slot
speedcodex/terraskipped · already recorded this slot
speedcodex/solskipped · already recorded this slot
speedcodex/gpt-5.5skipped · already recorded this slot
speedgrok/grok-4.6skipped · already recorded this slot
speedgrok/grok-4.5skipped · already recorded this slot
speedclaude/haikuskipped · already recorded this slot
speedclaude/sonnetskipped · already recorded this slot
Aug 23, 4:00:01 PMpulsescheduled1m 38s · 13 rec · 0 skip · 6 fail · 38.1k tok
TargetResultWallNew inCacheOutTotalAPI $
pinggrokok7.28s9,691256299,976$0.0197
pingclaudeok4.37s250078328$0.0006
pingcodexfailedYou've hit your usage limit. Visit https://chatgpt.com/codex/settings/usage to purchase more credits or try again at Aug 24th, 2026 8:30 PM.7.13s0000$0.0000
speedclaude/opusok6.84s3060241547$0.0076
speedclaude/fablerate limit warningserved with a rate limit warning: seven_day_overage_included at 0.75 utilization8.43s3060241547$0.0151
speedcodex/lunafailedYou've hit your usage limit. Visit https://chatgpt.com/codex/settings/usage to purchase more credits or try again at Aug 24th, 2026 8:30 PM.7.38s0000$0.0000
speedcodex/terrafailedYou've hit your usage limit. Visit https://chatgpt.com/codex/settings/usage to purchase more credits or try again at Aug 24th, 2026 8:30 PM.6.00s0000$0.0000
speedcodex/solfailedYou've hit your usage limit. Visit https://chatgpt.com/codex/settings/usage to purchase more credits or try again at Aug 24th, 2026 8:30 PM.7.38s0000$0.0000
speedcodex/gpt-5.5failedYou've hit your usage limit. Visit https://chatgpt.com/codex/settings/usage to purchase more credits or try again at Aug 24th, 2026 8:30 PM.6.32s0000$0.0000
speedgrok/grok-4.6ok6.63s12.1k25629312.7k$0.0262
speedgrok/grok-4.5ok6.54s12.1k25628512.7k$0.0261
speedclaude/haikuok9.44s2720550822$0.0030
speedclaude/sonnetok8.48s3220241563$0.0031
Aug 23, 3:00:00 PMpulsescheduled2m 3s · 13 rec · 0 skip · 0 fail · 99.2k tok
TargetResultWallNew inCacheOutTotalAPI $
pingcodexok8.52s12.0k0512.0k$0.0024
pinggrokok5.80s9,690256179,963$0.0196
pingclaudeok4.94s250073323$0.0006
speedclaude/sonnetok10.5s3220270592$0.0033
speedclaude/opusok7.25s3060241547$0.0076
speedclaude/fableok9.15s3070241548$0.0151
speedcodex/lunaok13.7s10.8k025511.0k$0.0025
speedcodex/terraok10.5s12.3k024312.6k$0.0276
speedcodex/solok12.4s12.3k024312.6k$0.0690
speedcodex/gpt-5.5ok13.0s6,9955,63225512.9k$0.0454
speedgrok/grok-4.6ok6.11s12.1k25629312.7k$0.0261
speedgrok/grok-4.5ok6.76s12.4k029212.7k$0.0265
speedclaude/haikuok9.69s2720554826$0.0030
Aug 23, 2:00:00 PMpulsescheduled1m 49s · 13 rec · 0 skip · 0 fail · 96.9k tok
TargetResultWallNew inCacheOutTotalAPI $
pingclaudeok5.14s2500106356$0.0008
pingcodexok6.40s10.8k0510.8k$0.0022
pinggrokok5.19s9,436512299,977$0.0193
speedclaude/haikuok8.54s2720546818$0.0030
speedclaude/sonnetok7.75s3220241563$0.0031
speedclaude/opusok7.24s3060241547$0.0076
speedclaude/fableok7.89s3070241548$0.0151
speedcodex/lunaok12.4s10.8k025511.0k$0.0025
speedcodex/terraok9.68s12.3k024312.6k$0.0276
speedcodex/solok10.9s12.3k024312.6k$0.0690
speedcodex/gpt-5.5ok10.7s11.4k025711.6k$0.0646
speedgrok/grok-4.6ok8.53s11.9k51228312.7k$0.0258
speedgrok/grok-4.5ok5.50s11.9k51228312.7k$0.0257

Each row: start time · kind · trigger (cron means the timer, anything else means by hand) · wall time · rec / skip / fail · total tokens. Skipped means it was not that target's turn, or a rate limit stopped the app. Failed is a real error, and we keep it.

Sample log

Every measurement we saved, newest first. Filter by check or by app. Most old test-kit rows stay in Logs only. Older speed rows that can be rebuilt exactly from their raw files may return to the history. The counts above always cover the full window, not the filter.

This window, counted by outcome

  • ok · 49
  • failed · 10
  • rate limit warning · 2
  • wrong answer · 1

Every saved sample in the window, sorted into its label. The filters below change the table, never these counts.

The status words

ok
The app answered and the answer passed our grader.
rate limit
The vendor refused the request. We stop the rest of that app for the round. Busy plan, not a slow model.
rate limit warning
The vendor served the correct answer but flagged that the plan's usage window is getting full. Its real wait stays in experienced-speed summaries, and the warning is labeled. It never stops the round.
timeout
The app never finished in the time we allow.
auth
The sign-in was not accepted. We skip that app until it is fixed.
used tools
The app reached for a tool even though we said not to. The row does not count.
wrong answer
Text came back, but it was not what we asked for. Speed with a wrong answer never counts.
remapped
We asked for one model and the app served another. We save both names and mark the row, and the boards treat it with care.
parse
The app’s own output was not readable, so we could not trust the timings.
overloaded / credits / failed
The vendor said it was busy, out of credit, or gave a plain error.
StartedCheckTargetStatusWallTotalCostAPI $
Aug 23, 5:01:21 PMspeedclaude/opusok4.80s547$0.0085$0.0076
Aug 23, 5:01:15 PMspeedclaude/sonnetok5.24s563$0.0040$0.0031
Aug 23, 5:01:09 PMspeedclaude/haikuok6.62s803$0.0039$0.0029
Aug 23, 5:01:03 PMspeedgrok/grok-4.5ok6.25s12.7k$0.0089$0.0261
Aug 23, 5:00:55 PMspeedgrok/grok-4.6ok7.25s12.7k$0.0045$0.0264
Aug 23, 5:00:49 PMspeedcodex/gpt-5.5You've hit your usage limit. Visit https://chatgpt.com/codex/settings/usage to purchase more credits or try again at Aug 24th, 2026 8:30 PM.failed6.33s0$0.0000
Aug 23, 5:00:43 PMspeedcodex/solYou've hit your usage limit. Visit https://chatgpt.com/codex/settings/usage to purchase more credits or try again at Aug 24th, 2026 8:30 PM.failed5.96s0$0.0000
Aug 23, 5:00:36 PMspeedcodex/terraYou've hit your usage limit. Visit https://chatgpt.com/codex/settings/usage to purchase more credits or try again at Aug 24th, 2026 8:30 PM.failed6.42s0$0.0000
Aug 23, 5:00:29 PMspeedcodex/lunaYou've hit your usage limit. Visit https://chatgpt.com/codex/settings/usage to purchase more credits or try again at Aug 24th, 2026 8:30 PM.failed7.09s0$0.0000
Aug 23, 5:00:23 PMspeedclaude/fableserved with a rate limit warning: seven_day_overage_included at 0.76 utilizationrate limit warning6.46s547$0.0161$0.0151
Aug 23, 5:00:16 PMpinggrok/grok-4.6ok6.61s9,976$0.0033$0.0197
Aug 23, 5:00:08 PMpingcodex/lunaYou've hit your usage limit. Visit https://chatgpt.com/codex/settings/usage to purchase more credits or try again at Aug 24th, 2026 8:30 PM.failed6.91s0$0.0000
Aug 23, 5:00:05 PMpingclaude/haikuok3.48s325$0.0016$0.0006
Aug 23, 4:01:30 PMspeedclaude/sonnetok8.48s563$0.0040$0.0031
Aug 23, 4:01:21 PMspeedclaude/haikuok9.44s822$0.0040$0.0030
Aug 23, 4:01:14 PMspeedgrok/grok-4.5ok6.54s12.7k$0.0089$0.0261
Aug 23, 4:01:08 PMspeedgrok/grok-4.6ok6.63s12.7k$0.0044$0.0262
Aug 23, 4:01:01 PMspeedcodex/gpt-5.5You've hit your usage limit. Visit https://chatgpt.com/codex/settings/usage to purchase more credits or try again at Aug 24th, 2026 8:30 PM.failed6.32s0$0.0000
Aug 23, 4:00:54 PMspeedcodex/solYou've hit your usage limit. Visit https://chatgpt.com/codex/settings/usage to purchase more credits or try again at Aug 24th, 2026 8:30 PM.failed7.38s0$0.0000
Aug 23, 4:00:48 PMspeedcodex/terraYou've hit your usage limit. Visit https://chatgpt.com/codex/settings/usage to purchase more credits or try again at Aug 24th, 2026 8:30 PM.failed6.00s0$0.0000
Aug 23, 4:00:40 PMspeedcodex/lunaYou've hit your usage limit. Visit https://chatgpt.com/codex/settings/usage to purchase more credits or try again at Aug 24th, 2026 8:30 PM.failed7.38s0$0.0000
Aug 23, 4:00:32 PMspeedclaude/fableserved with a rate limit warning: seven_day_overage_included at 0.75 utilizationrate limit warning8.43s547$0.0161$0.0151
Aug 23, 4:00:25 PMspeedclaude/opusok6.84s547$0.0085$0.0076
Aug 23, 4:00:17 PMpingcodex/lunaYou've hit your usage limit. Visit https://chatgpt.com/codex/settings/usage to purchase more credits or try again at Aug 24th, 2026 8:30 PM.failed7.13s0$0.0000
Aug 23, 4:00:13 PMpingclaude/haikuok4.37s328$0.0016$0.0006
Aug 23, 4:00:06 PMpinggrok/grok-4.6ok7.28s9,976$0.0033$0.0197
Aug 23, 3:01:54 PMspeedclaude/haikuok9.69s826$0.0040$0.0030
Aug 23, 3:01:47 PMspeedgrok/grok-4.5ok6.76s12.7k$0.0090$0.0265
Aug 23, 3:01:41 PMspeedgrok/grok-4.6ok6.11s12.7k$0.0044$0.0261
Aug 23, 3:01:28 PMspeedcodex/gpt-5.5ok13.0s12.9k$0.0454
Aug 23, 3:01:15 PMspeedcodex/solok12.4s12.6k$0.0690
Aug 23, 3:01:05 PMspeedcodex/terraok10.5s12.6k$0.0276
Aug 23, 3:00:51 PMspeedcodex/lunaok13.7s11.0k$0.0025
Aug 23, 3:00:42 PMspeedclaude/fableok9.15s548$0.0161$0.0151
Aug 23, 3:00:34 PMspeedclaude/opusok7.25s547$0.0085$0.0076
Aug 23, 3:00:24 PMspeedclaude/sonnetok10.5s592$0.0043$0.0033
Aug 23, 3:00:19 PMpingclaude/haikuok4.94s323$0.0016$0.0006
Aug 23, 3:00:13 PMpinggrok/grok-4.6ok5.80s9,963$0.0033$0.0196
Aug 23, 3:00:04 PMpingcodex/lunaok8.52s12.0k$0.0024
Aug 23, 2:01:44 PMspeedgrok/grok-4.5ok5.50s12.7k$0.0087$0.0257
Aug 23, 2:01:35 PMspeedgrok/grok-4.6ok8.53s12.7k$0.0044$0.0258
Aug 23, 2:01:25 PMspeedcodex/gpt-5.5ok10.7s11.6k$0.0646
Aug 23, 2:01:14 PMspeedcodex/solok10.9s12.6k$0.0690
Aug 23, 2:01:04 PMspeedcodex/terraok9.68s12.6k$0.0276
Aug 23, 2:00:52 PMspeedcodex/lunaok12.4s11.0k$0.0025
Aug 23, 2:00:43 PMspeedclaude/fableok7.89s548$0.0161$0.0151
Aug 23, 2:00:36 PMspeedclaude/opusok7.24s547$0.0086$0.0076
Aug 23, 2:00:28 PMspeedclaude/sonnetok7.75s563$0.0040$0.0031
Aug 23, 2:00:20 PMspeedclaude/haikuok8.54s818$0.0040$0.0030
Aug 23, 2:00:14 PMpinggrok/grok-4.6ok5.19s9,977$0.0033$0.0193
Aug 23, 2:00:08 PMpingcodex/lunaok6.40s10.8k$0.0022
Aug 23, 2:00:03 PMpingclaude/haikuok5.14s356$0.0017$0.0008
Aug 23, 1:01:47 PMspeedgrok/grok-4.6ok5.96s12.7k$0.0045$0.0262
Aug 23, 1:01:35 PMspeedcodex/gpt-5.5ok11.8s11.6k$0.0646
Aug 23, 1:01:19 PMspeedcodex/solok15.8s13.8k$0.0752
Aug 23, 1:01:09 PMspeedcodex/terraReading additional input from stdin...wrong answer10.0s12.5k$0.0267
Aug 23, 1:00:57 PMspeedcodex/lunaok12.8s11.0k$0.0025
Aug 23, 1:00:49 PMspeedclaude/fableok7.76s548$0.0161$0.0151
Aug 23, 1:00:42 PMspeedclaude/opusok6.97s547$0.0085$0.0076
Aug 23, 1:00:34 PMspeedclaude/sonnetok7.27s563$0.0040$0.0031
Aug 23, 1:00:28 PMspeedclaude/haikuok6.21s803$0.0039$0.0029
Aug 23, 1:00:21 PMspeedgrok/grok-4.5ok7.10s12.7k$0.0090$0.0265

62 samples in the window. Same rows as JSON: /api/probes · /api/speed · /api/runs

Read next