How we test

• PingEvery hour

  What we ask

  Reply with exactly: ok

  What it measures

  How long the app takes to wake up and answer at all.

  This is not a writing-speed test. Most of a ping is the app starting up.

• SpeedEvery hour, every pinned model

  What we ask

  Count from 1 to 120 in plain text on a single line, numbers separated by single spaces. Output nothing else.

  What it measures

  Two simple things: how fast the visible answer was written, and how long the whole check took.

  The hero shows the newest completed check. The boards below wait for 12 checks and show what was typical.

Token

A token is a small chunk of text, roughly four letters. Models read and write text one token at a time.

Writing speed (tok/s)

Only the answer you can see, divided by the full start-to-finish time. Higher is faster. Hidden thinking never gives a model extra credit.

Total wait

How long the whole command took, including app startup and hidden thinking. Lower is faster. The 24-hour tables rank this number because every model writes the same fixed answer.

Stream tok/s

How fast text appears once it starts flowing. Claude counts visible answer tokens from the first word to the last. Grok does the same when its stream gives us a window. Codex sends no token stream, so its column covers the whole turn and carries a *. We show this number but never rank on it.

TTFT (time to first word)

How long you stare at a blank screen before the first visible word shows up. Thinking does not count as a word. Lower is better. Codex leaves this blank because it does not stream.

Typical value (median)

For a 24-hour board we line the checks up and take the middle one, not the average. We wait for 12 completed checks before publishing it.

Likely range

The range under total wait is a distribution-free 95% interval for the true middle. Overlapping ranges mean the data does not support a confident winner yet.

Slots / n

How many finished checks sit behind a number. A small count moves a lot, so we always print it next to the value.

Status words like rate limit and wrong answer are spelled out on the Logs page. Test kit versions are spelled out on the Changelog.

Clean room

Every run starts in a new empty folder. No saved settings, no skills, no slash commands, no memory, no tools. Grok gets a throwaway home folder. If an app reaches for a tool anyway, the row fails and is labeled used tools.

Same job for everyone

Same prompts, same one-line system prompt — “Answer the user. Do not use tools. Do not add a preamble.” — and every model at the top of its own effort scale, so no app can quietly coast.

Paid sign-ins only

We run on plans we pay for, like any customer. If an API key is sitting in the environment, the run stops on purpose. A key would quietly move billing to the API and we would stop testing the paid app.

One try, fails stay in

We never retry a bad run and we never pick the nicer of two samples. Every fail is recorded once and labeled: rate limit, rate limit warning, timeout, auth, used tools, wrong answer, or parse. Dropping fails would make every tool look better than it really is.

Every round is graded

A speed round only counts if the answer is right. A plain script checks the count against the one correct answer — no human opinion, no AI judge. A fast wrong answer scores nothing, so no app can win by writing rubbish quickly.

Nothing gets blended

Write speed and the ping stay on separate boards. We never mash them into one winner score, because the trade-off is yours to make.

The models we pin

The exact names we ask for every round. If an app serves something else, we save the real name and mark the row remapped.

Featured on the boards

claude-fable-5 and claude-opus-5 in Claude Code, gpt-5.6-sol in Codex, and grok-4.6 in Grok Build. Every board leads with these four.

Also measured

claude-haiku-4-5, claude-sonnet-5, gpt-5.6-luna, gpt-5.6-terra, gpt-5.5, and grok-4.5. Same rounds, same rules. They live on the Other models page.

Effort

Every pin runs at the top of its own scale: xhigh for grok-4.6, high for everything else. grok-4.5 has no xhigh, so high is its honest ceiling.

Ping carriers

The ping rides on claude-haiku-4-5, gpt-5.6-luna, and grok-4.6. gpt-5.2 was retired on August 19, 2026 — OpenAI now refuses it on subscription accounts.

The number

Q is the dollar value of one full usage window, priced as if the same tokens had been bought at the vendor's own published API list price. We divide what we spent in a window by how far the gauge moved, and read off what 100% would be. The point is the line over time, not the level: if a vendor quietly reduces what 100% of a window buys, Q falls.

What this is measured on

One account per tool and one machine. Our local logs see only this PC, so Q is a lower bound — a drop can be our own blind spot, not a vendor change. The detector's thresholds are set for about one false alarm per provider per year, and a flagged change always says which of the two it looks like.

How we read it

Every plan we track reports whole percentages of the current window, and nothing finer. We pair that gauge against our own spend, priced to the cent against each vendor's published rate card. We never buy usage to calibrate. Every number comes from work the lab was doing anyway, plus a gauge reading at most once every 20 minutes — the endpoint's budget is shared with the owner's own usage command, so we never poll it harder.

Measuring it costs nothing

The whole method is passive: usage endpoints the vendors already publish, plus files the apps already write. No extra model call is made for this board, so measuring the quota does not spend the quota.

It is our plan, not your account

The gauge we read belongs to the accounts we pay for. We do not read your quota, we do not want to, and no figure here says anything about how much you have left.

Why the number is a floor

The gauge counts everything on the account, including work on other devices we cannot see and cannot price. That always pushes a naive estimate down. So we do not average: we look for stretches where the gauge moved and our own spend explains all of it, and read the plan's size off those. Where only a floor is available we write “≥ $X” and say the gauge has not moved enough for a point estimate. We never dress a floor up as a point.

“Calibrated interval”, not a confidence interval

The range beside a published number is an envelope calibrated against simulated data where the true answer was known. It is built to hold across the whole plausible range of usage we cannot see, not to be a sampling interval around a random draw. Calling it a confidence interval would claim a guarantee this method does not make. A lower-bound row gets no range at all — a floor has no upper edge.

How accurate is it?

In simulation across a 0–30% share of usage we cannot see, the envelope estimator's median bias ranges roughly −12% to +19%; near our assumed 15–20% share it is about −8% to +6%, with RMSE 5–13%. Simple averages read 10–20% low, which is why we do not use one. That contamination share is assumed, not measured. Passive accuracy improves with time: about ±5% after roughly 6 weeks for Codex, 12 weeks for Grok, and 6 months for Claude, where the off-machine share is largest.

The price basis

Standard tier, never batch or flex. Cache reads and writes at the published rates. Reasoning tokens billed as output, never added on top. Long-context tiers applied per request where a vendor has one. A model our price table does not know is never valued at $0 — it is excluded, counted, and named.

Claude: what the plan says

Anthropic publishes no dollar or token figure for Max 20x. Extra usage credits are opt-in and prepaid, and are billed at standard API rates — which is exactly why API dollars are the right unit here. Fable 5 may draw up to 50% of the weekly limit. That is a documented scope rule on the plan, not a promotion, and it does not expire.

Codex: what a credit is worth

1 credit = $0.04 of API list price, derived from OpenAI's own USD rate card — exact on 9 of 10 models, with one 0.4% rounding. OpenAI does not publish a weekly credit allowance, but 15,000 credits = $600 is the round figure our measured value sits within a few percent of. Note that OpenAI calls both its $100 (5×) and $200 (20×) tiers “ChatGPT Pro”; we hold the $100 tier.

Grok has two currencies

Grok's own per-run cost is an internal basis — 0.17× list for grok-4.6 and 0.34× for grok-4.5 — and we show list dollars, with the basis carried beside them. So grok-4.5 draws about 2× the pool per list dollar of uncached traffic, and roughly 1.5× on typical cached traffic; a Grok figure is always “at list price for the mix we saw”. The gauge also lags spend by about 18 minutes, and readings are corrected for that lag. xAI publishes no pool size in any unit, so the Grok figure on the board is our own measurement and nothing else.

“100%” means one window

Windows are keyed by their reset time, and each one is measured on its own. Codex re-anchors when a reset is redeemed, so a week with three redeemed resets is three windows, not one that looks three times bigger. Claude's weekly window is a fixed account anchor; its 5-hour window rolls from first activity. Grok's weekly pool is shared with Chat, Imagine, and Voice.

A dollar of any model counts the same

We assume the pool is drawn down in proportion to list dollars, whatever the model. OpenAI's published credit rate card says exactly this. For Claude, extra usage is billed at standard API rates. For Grok it is open. The data could not identify a per-model weight for any of the three, so this is assumed and stated, not measured, and it is re-tested monthly.

Grades, and when we stay quiet

A is a day the gauge moved a lot and we caught nearly every reading; B is enough to name a number. C and below publish a floor only. A row the estimator marked as a floor stays a floor whatever its grade. A day with no usage produces no new point, and the board shows the last good value with its age.

When a vendor changes something

A plan rename, a window that changes length, a reset time that moves, or a new price table is flagged that day and starts a fresh baseline. Those are different products, not the same product shrinking.

Three versions on every row

The test kit version says how the samples were taken. The method version says how the number was estimated. The price table version says which published prices the dollars use. History is never repriced: a new price table starts a new version, and old rows keep the one they were computed with.

One lab, our numbers

Every number comes from one machine on a normal home connection, measured by us. Use it to compare the three apps against each other. These are not official figures from Anthropic, OpenAI, or xAI.

We time the app, not just the model

Start-up, the app’s own hidden prompt, and its network path all count. That is on purpose — it is what you wait for. It also means a quick model inside a slow app looks slow here.

Codex model proof

Codex does not echo the model name in its run events. Before timing, we read the model catalog from the same logged-in ChatGPT account, require the exact pinned model ID, and pass that ID to the CLI. If either check fails, the run cannot publish a speed. Codex also carries a large built-in prompt, so its input is much larger than the short visible task.

Tokenizers differ

Each app counts its own tokens, so two apps can split the same visible text into slightly different token counts. That is why the live hero puts total wait beside tok/s, and why the 24-hour tables rank the fixed answer by total wait.

Grok leftovers

A clean Grok 4.5 ping answered ok in 8.9s with 7987 input tokens. Its inspect output still lists a Codex plugin row and one hook. We name those leftovers on the sample. No tool ran. grok-4.5-build is the same family as grok-4.5.

We read our meter, not yours

We do read the usage meter on our own accounts, and we price a full window at the vendor’s published API rates — that is the quota board. We cannot see your account, and nothing here predicts when you will run out. Your windows also differ: Claude and ChatGPT meter in 5-hour and weekly windows, and SuperGrok is one weekly pool shared with Chat, Imagine, and Voice.

• Start here

  A one-minute tour of the boards, in plain words.

• Lab health

  Is the lab fresh right now, and is it still signed in?

• Logs

  Every sample we saved, fails included, plus our token spend.

• Changelog

  Every time we changed a rule on this page.