Amazon published a revised guideline for software development engineers on 27 October that recommends examining teams at the 10th and 90th percentiles of AI adoption, while judging their work by longer-term outcomes. Terk Terkowitz, vice president and distinguished engineer at Worldwide Amazon Stores, wrote Amazon’s post introducing the memo, which addresses how engineers and their managers should use and assess AI tools.
Key points
- Amazon says token use, AI adoption rates and code activity can raise costs or complexity when treated as targets.
- The company recommends checking whether heavy AI users deliver more, rather than discounting usage measures altogether.
- Amazon closed an internal usage leaderboard in May after employees used AI excessively to improve their rankings, CNBC reported.
Amazon’s 10th and 90th percentile checks
Amazon says its engineering leaders revised the role guideline after considering how AI is changing software development. The memo urges engineers to learn and apply AI in their daily work, but says the standards for assessing their contribution remain the effects on customers, the business and their teams. It describes long-term outcomes as the preferred measure, even though those outcomes can take time to appear and can be difficult to separate from other changes.
That difficulty has led Amazon to use measures such as tokens consumed and the rate at which staff adopt AI tools, according to the memo. Code changes and reviews offer another way to count activity. Amazon calls these measures “deeply flawed” when used to assess teams or organisations because they are removed from the results the company wants. It says turning usage into a target could increase spending, while treating lines of code as a target could add complexity.
Amazon nevertheless says those measures correlate well with the outputs it values and expects that relationship to continue for some time. The memo recommends using them with judgement when considering careers and team performance. At the 10th percentile of AI adoption, it advises looking for teams that are struggling to use the tools effectively. At the 90th, it recommends checking whether teams that use AI heavily are increasing delivery to match.
The revised guideline leaves room for engineers whose tasks gain little from current tools. “If your work does not benefit from current AI tooling, that is a finding, not a failing,” Amazon says. The company says an engineer who can explain where the tools fall short is exercising the judgement it expects.
Amazon closed a usage leaderboard in May
Amazon circulated the memo internally in September before putting it on its website on 27 October. The company closed an internal leaderboard in May after finding that employees were using AI excessively to rise through its rankings, CNBC reported. The practice, called tokenmaxxing, involves consuming tokens — the units processed and generated by AI models — with little regard for the value of the output.
An Amazon spokesperson said the company had not imposed caps on employees’ token use, CNBC reported. Other companies have limited token consumption to restrain AI bills or moved some tasks to cheaper open-source models. Amazon’s memo instead advises engineers to consider usage and code activity alongside other evidence of their work, rather than making either measure a goal in itself.
The leaderboard gave employees a reason to increase a count that Amazon’s new guidance says can be costly when pursued for its own sake. The memo also says that teams making greater use of AI have increased productivity and delivery, while teams that have neither adopted the tools nor raised their output are falling behind. Both positions appear in its advice to examine adoption alongside results.
Brooker measures launches and production changes
Marc Brooker, a vice president and distinguished engineer at Amazon Web Services, said Amazon has looked at “feature velocity” to assess how AI affects engineering productivity. That includes the number of launches and successful changes made in production. The measure concerns work delivered rather than the quantity of tokens consumed or code generated.
Brooker described the decision to spend on AI tools as a return-on-investment question. For a problem important to customers or the business, he said, Amazon would commit tools, staff time and tokens. He also warned against spending tokens on work that does not matter to either. His account puts the choice of task alongside the choice of tool when assessing an engineer’s work.
Brooker said it can be difficult to tell in advance which problems warrant that investment. Amazon therefore needs “some appetite for experimentation”, he said.
