Why do companies keep trying to use AI/token usage as a performance metric?
Because they've lost (or never had) the ability to quantify success.
Put another way, it is the exact same reason that vibe coders exist and succeed until they fail. AI is not currently reliable and the people who tend to rely on it most, also let it replace their own judgement and faculties. And they let it get that way because of a few (sometimes big) early wins.
Tokenmaxxing is an obvious result of an obviously flawed system.
How is the system "obviously flawed?"
I've written up a number of examples of my struggles with AI at work. But it boils down to this; AI isn't actually "smart" in the human sense and the results are non-deterministic. This means that the 2 developers using identical prompts can wind up with WILDLY different tokens spent for the same problem. AND more than likely, the developer that spent more tokens ends up with the inferior solution.
Identical prompts. Different token usage. Disproportionate results.
So, if I'm a developer and I actually have some functional knowledge of how AI works and what is going on, then I already know that ANY management who attempts to tie performance to token usage is just admitting that they don't understand the problem space well enough to measure output and are too lazy to do so. The logical way to protect my livelihood? Tokenmaxxing.
Now, as to why the session that spent more tokens is likely to be the worse answer? That is assuming that both sessions are running the same base model. But, fundamentally, most modern AI is run by agents in a harness. The harness keeps the AI agent(s) running in a loop until it gets to answer or an obvious failure state. All the while the context grows.
The context here is the problem. You see, AI doesn't frequently doubt itself. Once it has put a bad idea into its context, it is going to either keep trying to re-prove that point, or iterate very slowly. And the longer this goes on, the more likely important requirements will be purged from the context or ignored in pursuit of the goal.
An example; Co-Pilot changed some code, breaking something critical a hot path (and the associated unit tests) and then boldly asserted it had found AND corrected a production issue. The issue hadn't been fixed and it created the issue, but as I tried to explain both how to fix the problem AND that it was caused by the AI itself, it took me 5 additional prompts and over 30 minutes to get AI to solve a problem I could have solved in 5 minutes. And it never abandoned the belief that it had FOUND a legitimate production bug.
The change I had requested was TRIVIAL. I've seen Co-Pilot and Claude handle requests like these in seconds, or a few minutes and usually they come away fine. But, even when AI ends up at a good solution, if it takes more than 1 minute, I typically go and dig into the session and find out what the AI is doing. And it is a pretty consistent set of failures; it forgot to import a package. It forgot to add a namespace. I tried to run a Linux command on my Windows machine. And in the same loop it can make the SAME mistakes repeatedly. The harness is pissing tokens down the drain while it bloats the context.
Without fail, the best solutions I've ever gotten from AI are the ones that completed the fastest and used the least tokens.
And any management worth their paychecks would appreciate the superior outcome and reduced AI costs over the developer using more tokens. Traditional performance metrics are still the superior metrics. But those require domain expertise to implement and evaluate. There is a smart way to integrate AI usage into the mix, but it would be additive. You would expect that an employee using more tokens is also achieving proportionately better results. So, if you see an employee with little to no token usage falling behind similar employees who are using AI more, then that would be a sign of a place to improve. And, by contrast, an employee that is doing well, but FAR exceeds their colleagues in token usage is either using AI inefficiently, or simply tokenmaxxing.
The eagle-eyed may notice that I just advocated for measuring token usage after claiming it is a faulty metric. My experience is that the token usage IS highly variable, but in the long run does tend toward some averages when not abused. For example, the number of times I get sub-1 minute response from Claude Code 4.8 is maybe 1 in 20. Most queries will take 3-15 minutes. Anything longer than that and I generally just stop the agent, review what it was doing, and based on that either spin up a new session and try again or edit the prompt to bypass some pitfalls it hit.
Comments
Post a Comment