How is AI changing what makes a good engineer?
AI makes output easier to produce, but an engineer's contribution is still hard to see. Some thoughts on recognising good work as a manager.
I've been discussing productivity a lot lately, mostly how we know someone is doing well now that agents do so much of the work. I still want to understand this for myself and as a manager I need to give my team a decent answer too. I don't want people spending time on things just because we count them.
Each company has its own setup, but over the last few years what I've usually had at hand is Jira tickets, commits, PRs and logged hours. They're cheap to collect and look objective so it's easy to start there. But they don't tell us much about what actually changed because of someone's work.
I went back to some books and articles on this. Beck and Orosz describe a chain from effort to output, then outcome and impact. I like that way of picturing it because you can lose something at each step, a week of effort might ship nothing or a feature ships and nobody uses it.
What I can expect from someone depends on what they had control over. Sometimes we can see the impact. Other times delivering what was agreed is all we can fairly ask because the outcome depended on decisions outside their control, and delivering that work isn't the same as opening a lot of PRs.
With agents doing more of the typing I put more weight on how someone decides what is worth doing and understands what they're shipping. Reviewing the result and taking responsibility for it are part of the work too. I still need to recognise the people helping others get there, even when their own output looks quiet.
Output tells us less now
An agent can open a PR in a few minutes now. Three field experiments found an estimated 26% increase in completed tasks with an AI coding assistant. So there is evidence for more work getting done, though that doesn't tell us what those tasks changed.
PR and ticket counts can still help me see where work is getting stuck, the problem is using them to judge people. Robert Austin wrote about partial measures in Measuring and Managing Performance in Organizations back in 1996. If I reward someone for opening more PRs then I shouldn't be surprised when that's what they spend their time doing.
Reviews and glue work
We spend a lot of time discussing whether agents or humans should review code. I still need someone to understand what's being shipped and take responsibility for it, with checks before release and monitoring in production. When looking at their contribution I want to know what their review helped change rather than how many reviews they did.
Helping others get their work done is part of what Tanya Reilly calls glue work. One engineer I've worked with spent an afternoon discussing a change and saved the team a lot of rework, and in the tools that was one review comment. He also pairs with others to deliver changes and the credit often goes to them. I want to recognise this type of work by highlighting it as a much-needed form of impact.
Impact and contribution
In Impact Evaluation in Practice, Gertler and his coauthors describe impact as the difference between what happened and what would have happened without an intervention. That's useful when looking at our own work too. If I work on reducing incidents and they were already going down I can't claim the whole drop came from my change.
And even if we can estimate the impact we still have to work out who contributed to it. In the example below someone fixed checkout retries while another team changed the payments API and traffic dropped after a sale. I would want to understand whether the fix helped and what evidence we have for that. Trying to give everyone a percentage of the result doesn't seem very useful to me, two people can both be essential and their contributions overlap.
Our own feeling isn't reliable either
In a small 2025 METR study, experienced open source developers took 19% longer with AI tools but believed they'd been 20% faster. In its 2026 update, METR said developers were likely getting more speedup now but the evidence for how much was weak.
That makes it hard to trust our own feeling of being productive. And the tools change so fast that whatever we think we know might not hold in six months, I wouldn't be surprised if it changed within a quarter.
When the expected benefit doesn't happen
Effort and impact don't always go together. High effort with low impact is the part of this chart I don't want to overlook when judging someone's work.
Say the team is asked to deliver a new API and they do, then nobody adopts it or it gets rejected completely. The benefit we built it for didn't happen and we still spent time on it. There may be maintenance work too. But I don't think the effort was completely lost if people learned something useful for next time. That learning doesn't automatically make it worth the cost, I just wouldn't judge a reasonable bet that failed in the same way as work that was never worth doing.
We can miss what people learned when all we look at is whether the API was used. Recognising the effort matters for motivation too and I'd like to write more about that separately.
I want someone to be able to explain what their work helped change and how they know, including when the result wasn't what we hoped for.
References
- Kent Beck and Gergely Orosz, Measuring developer productivity? A response to McKinsey, The Pragmatic Engineer (2023). Where the chain comes from.
- Kevin Zheyuan Cui and others, The Effects of Generative AI on High-Skilled Work: Evidence from Three Field Experiments with Software Developers (2025). The estimated 26% increase in completed tasks.
- Robert D. Austin, Measuring and Managing Performance in Organizations, Dorset House (1996). What happens when a partial measure is used to judge people.
- Tanya Reilly, Being Glue (2019). Where the name glue work comes from.
- Paul Gertler and others, Impact Evaluation in Practice, second edition, World Bank (2016). Impact as the difference from what would have happened anyway.
- METR, Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (2025) and We are Changing our Developer Productivity Experiment Design (2026).
Disclaimer: Most of the text above is mine, but I used AI to help write and improve sections of this post.