The idea of optimizing software delivery performance is not new and many have sought ways of doing it. One common-sense conclusion everyone seems to agree with is: to improve something, you must be able to define it, split it into key components, and then measure those. But from here onwards, opinions as to what to actually measure, and HOW vary: Is project management the key component, is it speed, or is it the quality of code? One team at Google has dedicated years of academic research to this endeavor and has managed to back their hypothesis with real data. The results of this research are the DORA Metrics.
DORA stands for DevOps Research and Assessment (now part of Google Cloud) and is the official name of the team that defined the metrics and surveyed over 31,000 engineering professionals on DevOps practices, over the course of 6 years, making it the longest-running academic project in the field. Each year, they document its evolution and compile a State of DevOps report.
DORA Metrics have become an industry standard of how good organizations are at delivering software effectively, and are very useful for tracking improvement over time.
Did we get any better in the last year? This is a question DevOps team leaders most likely have to ask themselves a lot. Understanding DORA Metrics will help them assess their current status, set goals to optimize their performance, and understand how to do it.
But this is by no means limited to them. As we’ll see in the following lines, the benefits of tracking DORA Metrics go well beyond team borders. Engineering leaders will once and for all be able to make the case for the business value of DevOps.
The goal of this article is to:
– Define what DORA Metrics are: what did the groundbreaking research find?
– Provide industry values for these metrics
– Show you the tools you have in place to help you measure them
The origins of the DORA Metrics go a bit further back, when its 3 frontrunners, Nicole Forsgren, Jez Humble, and Gene Kim, set out to answer a simple but powerful question: how can we apply technology to drive business value?
They argued that delivery performance can be a competitive edge in business and wanted to identify the proven best way to effectively measure and optimize it.
Between 2013 and 2017, they interviewed more than 2000 tech companies and released their findings in a book titled: Accelerate, The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations.
Accelerate identified 4 major metrics for measuring software delivery performance (you will find them under slightly different names in the book, but for clarity and consistency with the current DORA naming, we will use the below):
While LTTC and CFR measure the code quality, DF and MTTR are velocity metrics. If they are consistently tracked, and if steps are taken to improve them, together, they can help Dev Ops leaders boost their team’s performance and bring real business results.
How? DORA uses these metrics to segment performers into Low, Medium, High, and Elite performers.
Let’s take a closer look at what each of these means and what are the industry values for each of the performer types described above.
Lead time to changes (LTTC) is the amount of time between commit and release.
For elite performers: < one day
For high performers:1 day – 1 week
For medium performers: 1 week – 1 month
For low performers: 1 month- 6 months
Pro tip: Companies that can fix bugs or make improvements faster tend to be more successful overall than the ones that take 2 to 3 months. To decrease LTTC, include testing in the development process. Your testers are the ones that can teach your developers how to write and automate tests, so you can take out an additional step. To be fast, you have to eliminate bottlenecks.
Deployment Frequency (DF) indicates how often a team successfully releases software.
For elite performers: multiple deploys per day
For high performers: once per day – once per week
For medium performers: once per week – once per month
For low performers: once per month- once per 6 months
Pro tip: if you release more often, and in smaller chunks, you will spend less time figuring out where the problem is. To minimize this risk, you should ship one pull request or change, individually, at a time.
For larger teams, where that’s not an option, you can create release trains, and ship code during fixed intervals throughout the day.
Mean Time to Restore (MTTR) measures downtime – the time needed to recover and fix all issues introduced by a release.
For elite performers: < 1 hour
For high and medium performers: < 1 day
For low performers: 1 week – 1 month
Pro tip: It’s important to look at all the metrics together, and not just MTTR, so you don’t end up with ‘quick fixes’ that only aggravate the issue in the future. If releasing often is part of your team’s culture, so will fixing things quickly be. Use feature flags to add a toggle button to changes, so that in the case of a failure, you can quickly turn that feature off, and reduce your MTTR.
Change Failure Rate (CFR) shows the percentage of releases that lead to downtime, or serious issues.
For elite, high, and medium performers: 0-15%
For low performers: 46-60%
Pro tip: Looking at the change failure rate instead of the total number of failures, will eliminate the false impression that the number of failures decreases with the number of releases. The more often you release, and in small batches, the less serious and easy to fix the defects are. If possible, make sure the developer deploying is also involved in the production, so they can easily understand the change and the bug, and the team can learn from them. This can greatly reduce the risk of running into that specific issue again.
The 2019 Accelerate State of DevOps report shows that Elite teams are twice as likely to meet or exceed their organizational performance goals.
According to the report:
elite performers now represent 20% of survey respondents.
High performers represent 23%,
medium performers represent 44%,
and low performers only represent 12%.
You can take the DevOps quick check to see what type of performer you are and how you rate against industry benchmarks.
So what was so groundbreaking about the research? Well, for the first time in the engineering industry, it was able to collect thousands of real-life examples and data from engineers all across the globe and prove that:
Then, the last task at hand remains how to measure DORA, and this is where development analytics comes into play.
Before Development Analytics existed, you had to use a similar method to the one used for the research – ask your developers. Check Jira statuses, create reports, and spend daily standups and 1:1 asking about updates until you get the full picture. But engineering team managers are not (all) academics and have a ton of other things to think about so this was obviously a tiresome and inaccurate process, with flawed results.
But let’s imagine for a second that the DORA team could connect all the data sources of the people interviewed to one single tool and analyze their work. Not possible in this scenario, of course. But it’s exactly what development analytics can do for you.
The Waydev platform analyzes data from your CI/CD tools, and automatically tracks and displays DORA Metrics without you requiring to aggregate individual release data.
Let’s look at Greg’s team. Greg is the DevOps team lead and opens Waydev to get ready for a weekly check-in with his manager. His team is now a high performer and has made significant progress over the past 4 months from more medium performance values.
While the deployment frequency is that of an elite performer, with multiple deploys per day, and Lead time to change high (under a week), recovery time can be significantly improved.
There is more you can do to increase your visibility into your team’s work. DORA is a great starting point, but to truly understand your developers, you need to dig deeper.
The cycle time breakdown will give you industry benchmarks for each of the stages in the software development process: coding, pickup, review, and deployment. This can help you determine your team’s productivity to then set standards and best practices.
The Developer Summary report is the easiest way to observe work patterns and spot blockers or just get a condensed view of all core metrics. It might be that the engineer in question has a high percentage of low-risk commits but has not produced new code because he’s been helping others, maybe some new colleagues.
The PR Resolution report can help you identify the bottlenecks in your PR cycles over the course of the sprint. Start incorporating metrics from the PR resolution report into your retrospectives, and coaching opportunities will be easier to identify. It’ll be more obvious when processes need to be adjusted, and as a team, you can work towards a more collaborative, productive, and engaging code review process.
The Targets feature enables users to set custom targets for their developers, teams, and organizations. You can check in on your goals and see how much progress has been made.
The report provides a heat map of when your team is most active. Most engineers perform better when they are deeply immersed in their work. Understanding this will help you schedule meetings and other events around their schedule.
You can find a list of all available features here.
In the end, the real takeaway here is: Focus on your team and goals, not on the metrics. Empower them, and give them the tools they need – they will be the ones able to make any changes.
Metrics and tools help your developers understand how they’re doing and if they’re progressing. Instead of relying on hunches, and gut feelings, they will be able to visualize their progress, spot roadblocks, and pinpoint what they need to improve.
This will make them feel more satisfied with their own work, more motivated, and engaged.
If you want to find out more about how Waydev can help you, schedule a demo.
Ready to improve your teams' performance?
