Caterpillar Case Study

Since the beginning of 2019, Caterpillar has onboarded hundreds of software engineers to work on their new platform. As expected, bringing in a lot of developers from different companies proved to be challenging, as they needed to get them aligned very quickly in terms of the ways of working and getting them productive, while ensuring that they were producing good quality software.

To make this whole process easier, Caterpillar started defining and documenting best practices. From their software development playbook and Scrum best practices manual to product management related practices, they documented everything and trained their team members on the related topics.

The next challenge that appeared was monitoring how well engineers adhere to these best practices and ensuring continuous improvements for teams and individuals. They figured that a different toolset was needed.

They were using Azure DevOps, which generated a lot of metrics but had the disadvantage of being very granular, therefore giving Engineering Managers and Team Leads a difficult time discovering the insights pertaining to their teams.

For this reason, they started looking for reporting tools that could offer valuable insights to Team Leads, Engineering Managers, and the Division and Platform Leadership team.

After looking at a number of products, Caterpillar settled on Waydev. After the initial proof of concept, which turned out well, they trained 20 Engineering Managers on using Waydev, the best practices, and the overall expectations. Soon after, they continued the training and implementation with their Division Heads, Engineering Managers, and Team Leads.

“One of the most interesting things that happened is, the pandemic started impacting us in mid-March, in the middle of tough deadlines and enormous amounts of engineering effort. Everybody had to go home, and there was an immediate concern: How is this going to impact us?

Where Waydev helped us tremendously is that, not only were we able to import the history and understand behaviors, even through the Christmas period, as things slowed down, as it picked up again in January, February. How well things were running in March and then, what was happening at that point.

We saw a bit of fluctuation in the first couple of weeks in March, as everybody started adjusting and getting used to everybody else being at home. And then we noticed that productivity returned to normal, and actually started increasing again.

We typically look at the changes around the lines of code. It’s a relative measure, but it’s a good indication of the amount of work that’s happening on a team by team basis.”

“We also looked at how frequently people commit. We’re aiming for one commit per day, and the reason why we want them to commit frequently is that we’re looking for PRs to be created and reviewed.

We’re trying to get the whole team involved in the code review process, not just the Team Leads or single individuals, especially in the code being merged without reviews.

Ultimately, that’s how we ensure code quality. We understand that you can have your code quality standards, but the code quality is a team responsibility.

When you’re attempting to develop a team of peers, everybody needs to participate in each software development activity and the Scrum life cycle.

From that perspective, Waydev has given us a lot more insight and quick access to data that would otherwise not have been there.”

“We’re now in a position where we can, where we can monitor teams and individuals. We have data to recommend adjustments where we see behaviors that we’re trying to reinforce in the teams not being followed.

At the end of the day, it’s always true; you can’t manage what you can’t measure. We can now measure, and we can measure at the team and individual level.

It’s always interesting when you talk to an Engineering Manager, and you review the stats. You say ‘Yup, this is expected behavior,’ or ‘Oh, that’s interesting,’ or ‘I need to talk to that person.’ You’ll find out that you sometimes have really great coders but very little participation in the teams, and I don’t think that’s a surprise to any of us.

So now we can work with individuals’ interactions relative to the rest of the team, and where we would like to see this person improve over time, and set some goals and targets.”

“That is a great question. There are competing products out there, and what value a product adds to the day-to-day engineering management function is a big question. Since these products are relatively new in the marketplace, and there’s a great deal of learning that needs to happen around them, we were cost-sensitive.

The fact that the products are relatively new, we think that there’s a great opportunity to improve these products. What we were looking for is, as we learn more about the products, how responsive the vendor would be to new requests or changes. We have found Waydev to be extremely responsible for new feature requests, changes, and open to discussing existing features. That was probably one of the biggest factors for us and deciding to go with Waydev.”