*This interview is part from Stripe Atlas community forum.
Measuring productivity in design fields like engineering is much harder than in operational fields like support.
For a support agent, you can track inbound tickets, time to resolve, and customer satisfaction and quantify productivity at every step of the funnel. For an engineer, the good output is hard to distinguish from the bad output with raw metrics. A really good engineer might take twice as long and ship less code than a bad engineer. But this may 10x the performance of your system or make it very easy for the next engineer to make changes, which is well worth the cost
So the first question I ask people trying to track engineering productivity is ‘Why?’
If you’re trying to increase productivity, approach it like a system debugging the problem. Instrument the system, find the biggest bottleneck, fix it, and move onto the next one. If you’re far away from day-to-day coding, sit down with an engineer and ship some code into production. Once you see everything laid out in steps, you can reason about the problem better.
If you’re trying to identify low performers, then you need a good performance framework. If this is a skillset where you’re not an expert (say, machine learning for example), then try to find and talk to 3 expert IC’s and ask them how they measure performance in their peers. At Coinbase, we measure an engineer’s performance based on five criteria:
The specifics change at each level, but this gives us a frame of reference to examine the work someone does and compare it relative to others.
Measuring productivity at the org level (several hundred people) is very different from measuring it at the team level (5-10), which is different from measuring it at the individual level.
As we grew to 50+ people, we created shared frameworks for individual and team performance. We have a leveling rubric for individual performance tracking (see the answer above) and OKR systems for team and organization performance tracking. It’s important to have a shared framework so that you’re not comparing apples to oranges.
Engineering productivity is qualitative so we measure the impact on business value (dollars generated, customers acquired) or the closest proxy that we can track (customer happiness, engagement, uptime). Engineering metrics like commit velocity and story points can be useful, but give little insight into value creation, which is what productivity should strive to measure.