24% PRs isn't 24% more productivity. Lines of code isn't productivity, and neither is CLs landed. What's the feature velocity of the team? How much time is being spent on rollbacks, outage responses, etc.?

Here's a quick hack to triple your PRs landed: Land a PR, then land ANOTHER PR undoing that one when you realize it was full of bugs, then land the PR again once you realize management doesn't care about quality, they just care about the number of PRs landed.

I've also seen an increase in merged PRs, but it coincided with developers opening smaller PRs. In other words, AI made devs break work down more so they opened more PRs for the same work. That's still good, because smaller increments are better, but there' no actual increase in coding productivity, and it means the context-switching burden from review work went up e.g developers slowed down in a different area.

> In my case this clearly doesn’t translate to as much value for the organization, or rather, it’s hard to say, as many of those PRs were things I wouldn’t even have done without AI support.

That's not necessarily bad. It could be a sign of effective prioritization. If you're good at working on the most important thing, and suddenly find yourself 24% more productive, what extra are you working on? The things that wouldn't have quite made the cut before.

Given that Microsoft's overhead to actually landing features has, from what I've heard, long been more about dodging the cross-org and inter-org "guns" famously depicted in [0] and [1] than coding time... a 24% increase in merged PRs is massive!

[0] https://static.ma.nu/publications/images/2013.07.12_new_york...

[1] https://newsletter.pragmaticengineer.com/i/138252015/drawing...

I think number of features released to customers (not behind a feature flag or still being rolled out, but fully rolled out). And number of bug fixes (only those reported by customers).

Also just in general, customer satisfaction, acquisition, conversion, retention, etc.

Number of completed org-level roadmap items, org-level goals achievement rate, and so on.

I also think a good one would be seeing an increase in meeting estimation, like if project was estimated to take X days with Y devs, does the use of AI increased how often you met or beat those estimates in actual time/dev effort?

And you'd want to compare that against prior years, where no AI was used, within the same org, or try going 1 quarter without AI and another with and compare quarter to quarter.

Incredibly hard problem, but METR had a good method. They had people estimate how long a task would take (before knowing whether AI would be used), and then randomly assigned each task to “with AI” or “without AI.” When the data was in, they compared actual/estimate ratios of the two populations.

(Presumably, they used a t-test that only compared people against themselves.)

Interestingly, for that study (released in 2025), participants self-rated themselves as 20% more productive, but were measured as being 19% less productive.

Microsoft itself has a system for measuring this which outlines a few example metrics: https://queue.acm.org/detail.cfm?id=3819080

There was a nice talk about this by one of the author's at this year's BUILD conference: https://build.microsoft.com/en-US/sessions/BRK210

It'll probably take a really good product built by a profitable company evangelizing an AI workflow with reproducible examples dating back a few years.

We use a tool called Weave (I believe YC 25?) that analyzes PRs for "expert units of work" and shows lift from AI tools. My understanding is they have their own proprietary model that assesses the difficulty of each PR. I find the organization level view and pivots useful and aligned with intuitive expectations.

ultimately product success metrics such as market share and revenue, but those are lagging and might not be measurable over a 4 month period (the original study duration). beyond that, behaviour changes for users (is the product more useful?). those can be measured more frequently and can be compared to the previous period. if not that, than delivery output metrics such as cycle time for development (are we delivering faster?) paired with failure demand (% of tickets we thought were delivered but required rework/fixing). PRs are tied to effort/delivery process inputs, not outputs.

Engineering headcount.

It's wild how we need to rediscover Goodhart's Law every time we have a new thing we think might increase productivity.

Completely disable the usage of AI for time period t. Compare the impact on whatever before t and during t.

Exactly. Once AI-assisted PR count becomes a performance metric, the metric is no longer merely incomplete—it actively creates an incentive to generate more tasks and PRs