- What the test asks. Six development modules, their current answer checks, and what they do not establish.
- Why scores can lie, and what this does about it. Paired input changes aim to expose brittle answers, but templates and shortcuts remain a limitation.
- Which runs have qualified. No qualified results yet. The current task bank is still under review.
What has qualified so far
Only the newest qualifying record for each exact model and effort can appear. Failed, stale, synthetic or unapproved evidence is withheld.
No results to show yet
No approved numerical result is available in this snapshot. The current task bank remains under review.
The bar for a qualified result
- Reviewed method identity and hash from the committed readiness registry.
- Exact model, route and effort; newest attempt first; fresh evidence.
- All six domains complete, all planned tasks attempted, no invalid attempts.
- Passed validation, consistent counts and a reviewed private suite.
- Current policy approves no IntelliBench method, so numerical results remain withheld.
Practical work, six ways
Each module tests one skill with fresh task instances and a paired twin. Nothing here is scored — this is what the test asks, not who passed it.
Module 1 of 6
Code repair
Inspect a small code example.
- What it asks
- Development tasks use small code examples and expected text answers. They do not execute a repaired program or establish repository repair ability.
- The paired twin
- Some inputs or code details change between paired tasks. Template shortcuts may still work.
- What counts as a pass
- The answer must match its reference. No executable test-suite result is implied.
Current evidence status and any qualified pass counts appear in Results above.
What a paired task feels like
This example was written for this page so you can see the idea. It is not scored, not ranked, and not taken from the hidden task pool.
The task
Plan a two-day team offsite for 12 people. Budget: $4,000. Within 90 minutes of the office. The venue must seat 12 and allow outside food.
A passing plan names a real venue, stays under budget, fits the travel window, and meets both venue rules.
Its paired twin
Same brief, but the budget drops to $2,500 — and one attendee needs step-free access.
A plan copied from the first answer now breaks twice. Only re-planning from the new numbers passes. That is the whole idea: the twin rewards doing the work again, not remembering the answer.
Teaching example · made up for this page · not scored · not extracted from hidden instances. Real tasks stay private so models cannot rehearse them.
The score changed. Did the work?
Benchmaxxing means chasing a high test score without getting better at the work outside the test. Three things make that possible — and three defences push back.
How a score misleads
- Leaked tasks. The model saw the test questions during training, so it recites instead of reasoning.
- Narrow training. The model is drilled on the test's shape — good at that shape, brittle everywhere else.
- Cherry-picked setup. The best of many tries is reported, or the settings quietly favour one model.
What IntelliBench does about it
- Private rotating instances. Tasks are intended to remain private, with varied inputs. This can reduce simple rehearsal but does not prevent shortcuts or contamination.
- Paired changes. Every task has a twin with shifted details. A changed input can reveal an unchanged answer, but a twin alone does not prove robust reasoning.
- Deterministic checks. The current development bank compares answers with task references. It does not execute generated code or measure probability calibration.
The honest limits
These defences raise the bar; they do not make a test un-gameable, and we never claim that. Hidden tasks can still leak. Twins test robustness, not the whole of intelligence. Six modules do not cover every job. A high IntelliBench score is evidence about these practical tasks — not a universal intelligence ranking, and not proof that a model will do your work well.
How a result earns its place
Protocol intellibench-practical-v1. The Denominators are public; the task instances are not.
- The denominator is everything
- Score is 100 × passed ÷ planned. Failed answers stay in the denominator — a wrong answer is a miss, not a missing row. Attempts that fail for lab reasons (outages, identity failures) are marked invalid instead, and any invalid attempt disqualifies the whole run rather than quietly disappearing.
- Reviewed before it is shown
- Every run passes through planned, attempted, checked, reviewed and published. Only reviewed runs with full coverage and a written reason qualify. Research pilots and one-off smoke runs are labelled as such and never ranked.
- Same test or no comparison
- Each run carries its method version, suite fingerprint, effort and provider route. Runs on different versions, suites, efforts or routes are different tests — this page says so instead of ranking them together. There is no cross-table winner.
- Evidence without exposure
- Each run shows a fingerprint of its evidence, its dates, and its token and cost usage when reported — a dash when unknown. Full evidence stays with the lab and is never served by this site. Nothing on this page runs a benchmark or spends API budget on your visit.