Octopus Deploy logo

Octopus Deploy shipped 47% more PRs

A woman and two men looking at a computer outdoors, suggesting they are doing a code review together.
89%

reduction in feedback burden on one principal engineer

56%

increase in <code-text>Feedback Given<code-text>

47%

increase in <code-text>Merge Frequency<code-text>

Octopus Deploy – which makes complex deployments simple and consistent – used Multitudes to rebalance reviews across the team, to not overload one principal engineer, while keeping delivery strong.

Background

Engineering Manager Kim Engel used Multitudes insights to identify that a principal engineer was overloaded with reviews. Based on that, they decided to redistribute the feedback load across the team. This improved the team’s ability to deliver work – both by freeingup the principal engineer to do more planning work and because the team was able to complete more PRs.


The challenge

Kim’s team at Octopus Deploy was growing quickly – they had brought on several new team members. With all the new engineers onboarding, there were fewer people on the team who knew the codebase and could do reviews. Kim wanted to see how these changes were impacting the team dynamics and the work they did. She had a feeling that the more experienced developers were carrying the review burden, but wanted to get data to explore that further.

“I suspected it, but Multitudes gave me a datapoint. This way, change is not just based on how someone feels.”

Kim Engel, Engineering Manager - Cloud, Octopus Deploy


Our unique insight

Using metrics from the DORA research, Multitudes showed that the team had maintained a good pace throughout this period (their <code-text>Time to Merge<code-text> was low), but they were merging fewer PRs overall – their <code-text>Merge Frequency<code-text> was low and decreasing. At the same time, their <code-text>Change Failure Rate<code-text> had been increasing – meaning that there were more release failures that they were fixing.

Kim was particularly interested in the feedback dynamics that were impacting these bigger trends. When she dived into the collaboration data she could see that the <code-text>Feedback Given<code-text> was low and decreasing. In fact, the <code-text>Feedback Flows<code-text> data revealed that one principal engineer had been giving 42% of the feedback for the last 6 weeks.  

Kim knew that this principal engineer had many demands on their time and didn’t want to overload them with reviews. In addition, research shows the importance of reviews for knowledge-sharing across the team, so she wanted to make sure more people were participating in that sharing.

A line graph to illustrate Merge Frequency was reducing over the 6 weeks to mid-July 2022.
A line graph to illustrate the Change Failure Rate increasing over the 6 weeks to mid-July 2022.
A line graph to illustrate the PR Feedback Given decreasing over the 6 weeks to mid-July 2022.
A sankey diagram showing Feedback Flows, or the distribution of comments given and received on PRs.


Actions taken

Kim met with the team’s leadership group to share the insights and discuss next steps. They agreed that it was important to reduce the review burden on the principal engineer, both to support their wellbeing and so that they could have more time for planning work. 

Kim then had a conversation with the broader team where she shared the goal of getting more people across the team to jump in on reviews. They spoke about how it’s useful for everyone to do reviews, no matter their seniority level. In the team discussion, the team also decided that when anyone sees requests for PR reviews in Slack, they can just pick them up. This was made even easier when the team installed the Multitudes PR Slack Alerts, which flag PRs that have been blocked and need a review or that might be ready to be merged.


Outcome

The immediate impact was that the review burden on the principal engineer immediately reduced– they went from carrying 42% of the feedback load to just 5%. This freed up the principal engineer to do more future planning work, something they were uniquely able to do because of their experience.

At the same time, the team was incredibly successful at getting more feedback from across the team – total <code-text>Feedback Given<code-text> actually increased 56% over the 6 weeks following the change. 

Finally, all these improvements in how the team gives feedback were also correlated with an improvement in delivery – the team observed a 47% increase in PRs merged per person (<code-text>Merge Frequency<code-text>) over this same period, and their <code-text>Change Failure Rate<code-text> reversed its upward trend.

A sankey diagram showing Feedback Flows, or the distribution of comments given and received on PRs.
A line graph to illustrate the PR Feedback Given increasing over the following 6 weeks to mid-August 2022.
A line graph to illustrate the Change Failure Rate decreasing over the 6 weeks to mid-August 2022.
A line graph to illustrate Merge Frequency was increasing over the 6 weeks to mid-August 2022.

What next? 

To proactively catch future issues, Kim could set up <code-text>1:1 Prompts<code-text> in Slack. These alerts use behavioral data to automatically identify when someone is carrying a higher feedback load, not getting timely feedback, or is at risk of burnout and then suggest check-in questions for their next 1:1 conversation.

Two team members standing side-by-side, one looking at the text to the left.

Empower your engineering managers to build the best team.

See Multitudes in action
Two team members standing side-by-side, one looking at the text to the left.