Feuerfest

I don't remember when and where I read this statement. I just know that I was still in my training as an IT professional. Judging by that it must have been the late 2000s or early 2010s.

The statement was the following:

Every metric you monitor will improve.

This is a warning.

I don't know why I still remember it. Maybe because my final project for my vocational final examination was the design and set up of a monitoring system for my employer. And I still have some sort of affinity or inclination for the whole monitoring topic.

Only later when I started working at a big German telecommunications provider I understood the second part of this statement to its fullest.

When I, before, thought of meticulous colleagues deleting too many files to free up enough disk space in order to make that "disk usage" check displaying a bright green "OK" again, I started to learn of the wonderful world of Key Performance Indicators, or KPIs.

I saw how badly formulated bonus-goals led to some management folks cheating the system. Yes, the KPIs looked good on paper. The math did check out - so to speak. But did they achieve what the KPIs were really meant for? Yeah.. Not so much. Humans are humans, after all.

And then there is Amazon. It made the news yesterday that Amazon had an internal high score board of some kind to improve AI usage by employees. It was called Kirorank. This board wasn't based on professional evaluations by co-workers or the like. No, as it seems it was just based on token-consumption.

...

...

...

I think now we can all see where this is heading, right?

And this is exactly what happened. Employees created over-engineered AI tasks to consume as many tokens as possible. Tokens for which Amazon has to pay Anthropic (the AI models they are using).

Hence Amazon ditched Kirorank and switched to a new metric.

"According to the report, Amazon is now using a different metric called "normalised deployments" to evaluate the use of AI tools internally. Instead of token consumption, the metric measures how regularly developers use AI for meaningful code."
Source: https://www.heise.de/en/news/Too-much-tokenmaxxing-Amazon-stops-internal-AI-ranking-11311902.html

But why was Kirorank created in the first place? Well, taking a quote from the Heise-article:

The leaderboard was created by a group of employees who wanted to drive awareness for how AI can accelerate work.

And that reminds me of another piece of wisdom. 😅

The road to hell is paved with good intentions.