How is AI changing what makes a good engineer?

AI makes output easier to produce, but an engineer's contribution is still hard to see. Some thoughts on recognising good work as a manager.

I've been discussing productivity a lot lately, mostly how we know someone is doing well now that agents do so much of the work. I still want to understand this for myself and as a manager I need to give my team a decent answer too. I don't want people spending time on things just because we count them.

Each company has its own setup, but over the last few years what I've usually had at hand is Jira tickets, commits, PRs and logged hours. They're cheap to collect and look objective so it's easy to start there. But they don't tell us much about what actually changed because of someone's work.

I went back to some books and articles on this. Beck and Orosz describe a chain from effort to output, then outcome and impact. I like that way of picturing it because you can lose something at each step, a week of effort might ship nothing or a feature ships and nobody uses it.

Four connected boxes with icons. Effort: work put in, with a clock. Output: what we delivered, with code brackets. Outcome: what changed, with a checkmark. Impact: what changed because of it, with a rising chart. Effort and output are easier to count. Outcome and impact are harder to establish. Red arrows show nothing delivered, output nobody uses and what would have happened anyway. Notes say AI can speed up delivery, while results can be slow to show up and hard to credit to one person.
Effort, output, outcome and impact, and what gets lost on the way.

What I can expect from someone depends on what they had control over. Sometimes we can see the impact. Other times delivering what was agreed is all we can fairly ask because the outcome depended on decisions outside their control, and delivering that work isn't the same as opening a lot of PRs.

With agents doing more of the typing I put more weight on how someone decides what is worth doing and understands what they're shipping. Reviewing the result and taking responsibility for it are part of the work too. I still need to recognise the people helping others get there, even when their own output looks quiet.

Output tells us less now

An agent can open a PR in a few minutes now. Three field experiments found an estimated 26% increase in completed tasks with an AI coding assistant. So there is evidence for more work getting done, though that doesn't tell us what those tasks changed.

PR and ticket counts can still help me see where work is getting stuck, the problem is using them to judge people. Robert Austin wrote about partial measures in Measuring and Managing Performance in Organizations back in 1996. If I reward someone for opening more PRs then I shouldn't be surprised when that's what they spend their time doing.

Reviews and glue work

We spend a lot of time discussing whether agents or humans should review code. I still need someone to understand what's being shipped and take responsibility for it, with checks before release and monitoring in production. When looking at their contribution I want to know what their review helped change rather than how many reviews they did.

Helping others get their work done is part of what Tanya Reilly calls glue work. One engineer I've worked with spent an afternoon discussing a change and saved the team a lot of rework, and in the tools that was one review comment. He also pairs with others to deliver changes and the credit often goes to them. I want to recognise this type of work by highlighting it as a much-needed form of impact.

Impact and contribution

In Impact Evaluation in Practice, Gertler and his coauthors describe impact as the difference between what happened and what would have happened without an intervention. That's useful when looking at our own work too. If I work on reducing incidents and they were already going down I can't claim the whole drop came from my change.

A line chart of incidents per month over time. Incidents are already falling before a vertical line marked Change released. After it the solid line, Observed incidents, falls faster. A dashed line, Without the change, keeps falling at the old rate. The gap between the lines is labelled Estimated impact.
A made-up example. The gap between the two lines is the estimated impact.

And even if we can estimate the impact we still have to work out who contributed to it. In the example below someone fixed checkout retries while another team changed the payments API and traffic dropped after a sale. I would want to understand whether the fix helped and what evidence we have for that. Trying to give everyone a percentage of the result doesn't seem very useful to me, two people can both be essential and their contributions overlap.

Several causes feed the same result of checkout errors down 30%: your checkout retry fix, another team's payments API update, lower traffic and a teammate's improved failure recovery. The diagram asks Did the fix help?, with evidence for the change and other causes considered. No share of the result is assigned to any one cause.
A made-up example with several causes behind the same result.

Our own feeling isn't reliable either

In a small 2025 METR study, experienced open source developers took 19% longer with AI tools but believed they'd been 20% faster. In its 2026 update, METR said developers were likely getting more speedup now but the evidence for how much was weak.

That makes it hard to trust our own feeling of being productive. And the tools change so fast that whatever we think we know might not hold in six months, I wouldn't be surprised if it changed within a quarter.

When the expected benefit doesn't happen

Effort and impact don't always go together. High effort with low impact is the part of this chart I don't want to overlook when judging someone's work.

A chart with effort on the horizontal axis and impact on the vertical axis, split in four. Low effort, high impact: Leverage, small fix with a large benefit. High effort, high impact: Big bets that paid off, a migration reduced incidents. Low effort, low impact: Small changes, minor cleanup. High effort, low impact: Bets that didn't land, an API delivered with no adoption. A note says Learning may remain.
The examples are made up. High effort with low impact also includes reasonable bets that didn't pay off.

Say the team is asked to deliver a new API and they do, then nobody adopts it or it gets rejected completely. The benefit we built it for didn't happen and we still spent time on it. There may be maintenance work too. But I don't think the effort was completely lost if people learned something useful for next time. That learning doesn't automatically make it worth the cost, I just wouldn't judge a reasonable bet that failed in the same way as work that was never worth doing.

We can miss what people learned when all we look at is whether the API was used. Recognising the effort matters for motivation too and I'd like to write more about that separately.

I want someone to be able to explain what their work helped change and how they know, including when the result wasn't what we hoped for.

References

Disclaimer: Most of the text above is mine, but I used AI to help write and improve sections of this post.