The token-efficiency gap that wasn’t

The problem with 70-token problems

The original benchmark asked agents to solve Rosetta Code exercises. The winning solution averaged 70 tokens in J and 109 in Clojure. A task that fits in 70 tokens is not a task — it is a print statement with extra steps, and almost all of the measured “cost” is the shape of the answer rather than the work of getting there.

So Luu re-ran the question on two problems with real surface area. First: read the Zstandard RFC and implement a complete decoder, with no access to the tests. Each language got 40 runs at each of two effort levels (Swift and the two assembly targets got 20), scored against 34 held-back test cases.

At medium effort, the original claim looks alive. The dynamic languages cluster up and to the left — cheaper and more correct:

Python is the cheapest and the most correct. C++ is right beside it, but the four cheapest points are all dynamic. If you stopped here — and the original benchmark effectively did — you would publish the same headline.

Then you turn the effort up

The same 25 languages, the same task, one knob moved. The clusters interleave, and the language on top is F#, which is static, statically inferred, and functional — roughly the opposite of the prediction:

Two things happened at once. The spread between best and worst language collapsed from 20 percentage points to 12, and the ordering scrambled. Whatever the medium-effort chart was measuring, it was not a durable property of the languages.

The dead heat

Grouping the languages the way the original claim does — static versus dynamic — the 2.6× gap has to show up somewhere. It does not show up anywhere.

The two cuts matter, and it is worth being explicit about them, because choosing one is exactly how the original result got made. Across all 25 languages static leads by 2.2 points of correctness at medium effort — but that lead is manufactured by J and Factor, two obscure dynamic languages that do badly for reasons having nothing to do with dynamic typing. Drop them and it inverts to 0.1 points.

Cost behaves the same way, and cost is what the original claim was actually about. Among mainstream languages at medium effort, dynamic languages run 12.9% cheaper than static ones. Put the obscure languages back in and dynamic becomes 9.4% more expensive. The sign of the effect depends on which languages you decide to count, which is the signature of no effect at all.

And the 2.6× spread itself? It is real. The gap between the cheapest and priciest language on Zstd at medium effort is 2.99× — wider than the original claim. It simply has nothing to do with typing discipline. The two ends are Ruby at $3.33 and Factor at $9.96: a mainstream language against an obscure one. Keep only mainstream languages and the spread falls to 1.94×; turn the effort up and it falls to 1.88× across all 25. What the original benchmark caught was the cost of being unusual, mislabelled as the cost of being static.

The density thesis, inverted

The original post’s most striking result was J at 70 tokens, “nearly half of Clojure,” offered as a glimpse of how languages might evolve if token efficiency became the driver. On a real task, the density thesis inverts completely:

J costs $31.04 per run against Go’s $16.89, takes 58 minutes against 36, and finishes 25th of 25 on correctness. Only Factor, the other obscure language here, costs more. Terseness on the page is not cheapness in practice: a model that has seen little J has to think much harder to write it, and thinking is the thing you are paying for. Luu’s reading is that AI labs spend little or no synthetic-RL effort on obscure languages — and that popularity, not density, is the weak signal that actually shows up.

A second task, in case the first was a fluke

Zstd is a spec-implementation task. To check whether any of this generalises, the second eval is shaped differently: agents get the Pandoc ProgramBench materials and its tests, then are scored against a holdout set they never see. Scores are far lower — nobody gets past 31% — and the static/dynamic pattern is once again absent.

The top of this chart is a coin flip between Python, Rust, PHP, and Ruby. The bottom is not a coin flip at all — assembly scores around 11% at 2.5× the cost. The reliable finding across both tasks is not about type systems. It is that weird is expensive.

What a language score is actually made of

Here is the part that never appears in a scatterplot. Each Zstd run reports exactly which of the 34 test cases it failed, so the per-language averages can be taken apart. Every row below is a language; every column is a test case.

The structure is stark. Five of the 34 cases defeat almost every language almost every time — compressed Huffman literals, multiblock text, the interop streams. Those five bands are doing most of the work in every average on this page. At the other end, five cases are passed by essentially everyone. The middle is thin.

Now look at Clojure’s row. Besides the universal bands, it has two dark cells that are blank for all 24 other languages: expanded-rfc--rle-small and expanded-rfc--concatenated. Clojure fails each in 90% of runs. Every other language fails them in 0%.

Luu found the cause: Clojure’s byte conversion throws on values 128–255, the model reaches for it anyway, and the resulting programs die on any byte with the high bit set. He attributes 36 of 40 medium-effort Clojure failures to it. That single reflex is most of why Clojure sits last at medium effort — and why it climbs to mid-pack on Pandoc, a task where you touch far fewer raw bytes.

If we dig into why any particular condition got a certain score, the failures that caused the score are generally something idiosyncratic where it’s not always obvious how much the issue generalizes across tasks or across setups.

What survives

Two tasks cannot settle which language is best for coding agents, and Luu is explicit that they do not. But refuting a universal claim is cheaper than establishing one, and several popular claims do not survive contact with either task:

• Dynamic languages are 2–3× more token-efficient than static ones
  Not supported
• Dense languages like J point the way to token-efficient programming
  Inverted — J is 25th of 25 on Zstd, at nearly the highest cost
• Languages with lots of bad training data (PHP) do worse
  Not supported — PHP is mid-pack on both tasks
• You should use a popular language
  Weakly supported on both tasks

The deeper result is methodological. The first benchmark was not fraudulent; it measured something real about 70-token programs, and 70-token programs simply do not predict anything. Push the same question onto a task with real surface area and the effect does not shrink politely — it disappears, reverses, and turns out to have been a handful of idiosyncratic bugs wearing a trend’s clothing.

What makes this newly answerable is that the experiment costs about $20 per language per condition instead of a research grant. The questions that were unanswerable for decades — does this type system pay for itself, does this language cost more to maintain — are now merely expensive. That is a real change, and it mostly means we should expect more confidently-wrong benchmarks, not fewer.