A Summary of All of the State of DevOps Reports

It’s not that easy to find all the annual state of DevOps reports, partly because they forked in 2017 between Puppet and Google/DORA. Below I’ve listed each report by year, and I’m in the process of listing all the key findings from each report. Some reports provide greater insights than others.

The first report was in 2013, and showed quite clearly that adopting DevOps practices resulted in technological and business improvements. Along the way, Puppet and Google / DORA joined forces, parted ways, and now (as of writing in 2021) there are two State of DevOps Reports, and the focus has broadened to SRE, Organisational Culture, Security, and even Documentation.

2013 – Puppet:

  1. Respondents from organisations that implemented DevOps reported improved software deployment quality and more frequent software releases.
  2. DevOps enables high performance by increasing agility and reliability. High performing organisations ship code 30x faster and complete those deployments 8,000 times faster than their peers. They also have 50% fewer failures and restore service 12 times faster than their peers.
  3. Organisations that have implemented DevOps practices are up to five times more likely to be high-performing than those that have not. In fact, the longer organisations have been using DevOps practices, the better their performance: The best are getting better.

2014 – Puppet and DORA –

  1. Strong IT performance is a competitive advantage. Firms with high-performing IT organisations were twice as likely to exceed their profitability, market share and productivity goals.
  2. DevOps practices improve IT performance. IT performance strongly correlates with well-known DevOps practices such as use of version control and continuous delivery.
  3. Organizational culture matters. Organizational culture is one of the strongest predictors of both IT performance and overall performance of the organisation. High-trust organisations encourage good information flow, cross-functional collaboration, shared responsibilities, learning from failures and new ideas; they are also the most likely to perform at a high level.
  4. Job satisfaction is the No. 1 predictor of organisational performance. Job satisfaction includes doing work that’s challenging and meaningful, and being empowered to exercise skills and judgment. Where there is job satisfaction, employees bring the best of themselves to work: their engagement, their creativity and their strongest thinking.

2015 – Puppet and DORA:

  1. High-performing IT organisations deploy 30x more frequently with 200x shorter lead times; they have 60x fewer failures and recover 168x faster. Failures are unavoidable, but how quickly you detect and recover from failure can mean the difference between leading the market and struggling to catch up with the competition.
  2. Lean management and continuous delivery practices create the conditions for delivering value faster, sustainably.  This results in higher quality, shorter cycle times with quicker feedback loops, and lower costs. These practices also contribute to creating a culture of learning and continuous improvement.
  3. High performance is achievable whether your apps are greenfield, brownfield or legacy. As long as systems are architected with testability and deployability in mind, high performance is achievable.
  4. IT managers play a critical role in any DevOps transformation. Managers can do a lot to improve their team’s performance by ensuring work is not wasted
    and by investing in developing the capabilities of their people.
  5. Diversity matters. Research shows that teams with more women members have higher collective intelligence and achieve better business outcomes.
  6. Deployment pain can tell you a lot about your IT performance. Where code deployments are most painful, you’ll find the poorest IT performance, organisational performance and culture.
  7. Burnout can be prevented, and DevOps can help. Burnout is associated with pathological cultures and unproductive, wasteful work.

2016 – Puppet and DORA:

  1. High-performing organisations are decisively outperforming their lower-performing peers in terms of throughput. High performers deploy 200 times more frequently than low performers, with 2,555 times faster lead times. They also continue to significantly outperform low performers, with 24 times faster recovery times and three times lower change failure rates.
  2. High performers have better employee loyalty, as measured by employee Net Promoter Score (eNPS). Employees in high-performing organisations were 2.2 times more likely to recommend their organisation to a friend as a great place to work, and 1.8 times more likely to recommend their team to a friend as a great working environment. Other studies have shown that this is correlated with better business outcomes.
  3. Improving quality is everyone’s job. High-performing organisations spend 22 percent less time on unplanned work and rework. As a result, they are able to spend 29 percent more time on new work, such as new features or code. They are able to do this because they build quality into each stage of the development process through the use of continuous delivery practices, instead of retrofitting quality at the end of a development cycle.
  4. High performers spend 50 percent less time remediating security issues than low performers. Through better integrating information security objectives into daily work, teams achieve higher levels of IT performance and build more secure systems. less time on unplanned work and rework.
  5. Taking an experimental approach to product development can improve your IT and organisational performance. The product development cycle starts long before a developer starts coding. Your product team’s ability to decompose products and features into small batches; provide visibility into the flow of work from idea to production; and gather customer feedback to iterate and improve will predict both IT performance and deployment pain.

2017 – Puppet and DORA:

  1. Transformational leaders share five common characteristics that significantly shape an organisation’s culture and practices, leading to high performance. The characteristics of transformational leadership — vision, inspirational communication, intellectual stimulation, supportive leadership, and personal recognition — are highly correlated with IT performance.
  2. High-performing teams continue to achieve both faster throughput and better stability. The gap between high and low performers narrowed for throughput measures, as low performers reported improved deployment frequency and lead time for changes, compared to last year. However, the low performers reported slower recovery times and higher failure rates. It’s possible that pressure to deploy faster and more often causes lower performers to pay insufficient attention to building in quality.
  3. Automation is a huge boon to organisations. High performers automate significantly more of their configuration management, testing, deployments and change approval processes than other teams. The result is more time for innovation and a faster feedback cycle.
  4. Loosely coupled architectures and teams are the strongest predictor of continuous delivery. If you want to achieve higher IT performance, start shifting to loosely coupled services — services that can be developed and released independently of each other — and loosely coupled teams, which are empowered to make changes.
  5. Lean product management drives higher organisational performance. Lean product management practices help teams ship features that customers actually want, more frequently. This faster delivery cycle lets teams experiment, creating a feedback loop with customers.

2018 – Puppet:

  1. DevOps drives business growth – maintaining a robust software delivery and operability function increases productivity, profitability, and market share.
  2. Cloud technology correlates with business performance – this is enabled by reliable and sustainable cloud infrastructure, utilised via cloud native patterns.
  3. Open source software improves performance – high-performing IT teams are 1.75 times more likely to use open-source applications.
  4. Functional outsourcing can be detrimental to software performance, and Elite Performers are rarely using it.
  5. Technical practices such as monitoring and observability, continuous testing, database change management, and the early integration of security in software development all enable organisational performance.
  6. DORA identified high-performing organisations in a range of profit, not-for-profit, regulated, and non-regulated industries. The industry you’re in doesn’t affect your ability to perform.
  7. Diversity in tech is poor, but improving, and teams with improved diversity demonstrate higher performance than those that don’t.

2018 – DORA  (Accelerate):

  1. SDO (Software Delivery Organisation – i.e. development teams) performance unlocks competitive advantages. Those include increased profitability, productivity, market share, customer satisfaction, and the ability to achieve organisation and mission goals.
  2. How you implement cloud infrastructure matters. Proper (effective) usage of the public cloud improves software delivery performance and teams that leverage all of cloud computing’s essential characteristics are 23 times more likely to be high performers.
  3. Open source software improves performance. Open source software is 1.75 times more likely to be extensively used by the highest performers.
  4. Outsourcing by function is rarely adopted by elite performers and hurts performance. While outsourcing can save money, low-performing teams are almost 4 times as likely to outsource whole functions such as testing or operations than their highest-performing counterparts.
  5. Key technical practices drive high performance. These include monitoring and observability, continuous testing, database change management, and integrating security earlier in the SDLC.
  6. Industry doesn’t matter when it comes to achieving high performance for software delivery. High performers exist in both non-regulated and highly regulated industries alike.

2019 – Puppet:

  1. Doing DevOps well enables you to do security well.
  2. Integrating security deeply into the software delivery lifecycle makes teams more than twice as confident of their security posture.
  3. Integrating security throughout the software delivery lifecycle leads to positive outcomes.
  4. Security integration is messy, especially in the middle stages of evolution.

2019 – Google:

  1. The industry continues to improve, particularly among the elite performers.
  2. The best strategies for scaling DevOps in organisations focus on structural solutions that build community, including Communities of Practice.
  3. Cloud continues to be a differentiator for elite performers and drives high performance.
  4. To support productivity, organisations can foster a culture of psychological safety and make smart investments in tooling, information search, and reducing technical debt through flexible, extensible, and viewable systems.
  5. Heavyweight change approval processes, such as change approval boards, negatively impact speed and stability. In contrast, having a clearly understood process for changes drives speed and stability, as well as reductions in burnout.

2020 – Puppet:

  1. The industry still has a long way to go and there remain significant areas for improvement across all sectors.
  2. Internal platforms and platform teams are a key enabler of performance, and more organisations are adopting this approach.
  3. Adopting a product approach over project-oriented improves performance and facilitates improved adoption of DevOps cultures and practices.
  4. Lean, automated, and people-oriented change management processes improve velocity and performance.

2021 – Puppet:

  1. Organisational dynamics must be considered crucial to transformation.
  2. Cloud-native approaches are critical. It is no good to simply move traditional workloads to the cloud.
  3. Shift security, compliance and change governance left, and include security stakeholders in all stages of value delivery.
  4. Culture change is key, and must be promoted from the very “top” as well as delivered from the “bottom”. Psychological safety is at the core of digital and cultural transformations.

2021 – Accelerate:

  1. The “highest performers” continue to improve the velocity of delivery.
  2. Adoption of SRE practices improves wider organisational performance.
  3. Adoption of cloud technology accelerates software delivery and organisational performance. Multi-cloud adoption is also on the increase.
  4. Secure Software Supply Chains enable teams to deliver secure software quickly, safely and reliably.
  5. Documentation is important to being able to implement technical practices, make changes, and recover from incidents. 
  6. Inclusive and generative team cultures improve resilience and performance.

 

The Puppet State of DevOps Report 2021 – A Summary

I get a bit confused every year about who is writing the State of DevOps Report, and how that gets decided, and in the past it’s been Puppet, Google, DORA and others, but this year, 2021, it was definitely Puppet.

[Edit: apparently there are two State of DevOps reports now… I’m staying out of that particular argument though!]

The state of DevOps report each year attempts to synthesise and aggregate the current state of the technology industry across the world in respect to our collective transformation towards delivering value faster and more reliably. Or as Jonathan Smart puts it, “Sooner. Safer, Happier”. The DevOps shift has been in progress for over a decade now, and whilst DevOps was always really about culture, the most recent reports are now emphasising the importance of culture, progressive leadership, inclusion, and diversity more than ever before.

Last year, in 2020, the core findings of the State of DevOps Report focussed on:

  1. The technology industry in general still had a long way to go and there remained significant areas for improvement across all sectors.
  2. Internal platforms and platform teams are a key enabler of performance, and more organisations were starting to adopt this approach.
  3. Adopting a long-term product approach over short-term project-oriented improves performance and facilitates improved adoption of DevOps cultures and practices.
  4. Lean, automated, and people-oriented change management processes improve velocity and performance over traditional gated approaches.

 

This year (2021), there are a number of key findings building on previous DevOps reports:

1. Well defined and architected Team Topologies improve flow.

Clear organisational dynamics including well-defined boundaries, responsibilities, and interactions, are critical to achieve fast flow of of value. Whilst last year highlighted the importance of internal platforms, this report emphasises the importance of Conway’s Law, and shows that well defined team structures and interactions, such as platform teams (which scale out the benefits of DevOps transformations across multiple teams), cross-functional value-stream aligned teams, and enabling teams strongly influence the architecture and performance of the technology they build. Team “Interaction Modes” as seen in the diagram below are also critical to define, in the same way that we would define API specifications.

DevOps and Team Topologies

The book Team Topologies expands upon this concept in great detail, and Matthew and Manuel, the authors, also provide excellent training in order to apply these concepts to your contexts.

Clear team responsibilities

What is also clear from the State of DevOps Report this, and has been for some time, is that siloing DevOps practices into separate “DevOps teams” is an antipattern to success in most cases. And there should still be no such thing as a “DevOps Engineer”.

2. Use of cloud technology remains immature in many organisations.

Whilst the majority of organisations are now using cloud technology such as IaaS (infrastructure-as-a-service), most organisations are still using it in ways that are analogous to the ways we used to manage on-premise or datacentre technology. High performers are adopting “cloud-native” technologies and ways of working, including the NIST (National Institute of Standards and Technology)essential characteristics of cloud computing: “on-demand self-service, broad network access, resource pooling, rapid elasticity or expansion, and measured service.” How these are implemented is very context-specific, but includes the principles of platform(s) as a product or service, and high competencies in monitoring and alerting and SRE (Site Reliability Engineering) capabilities, whether as SRE teams, or SRE roles in cross-functional teams.

Cloud native capabilities and devops

3. Security is shifting left.

High performers in the technology space integrate security requirements early in the value chain, including security stakeholders into the design phase and build phases rather than just at deploy, or even worse, run, phases. Traditional “inspection” approaches to security, governance and compliance significantly impact flow and quality, resulting in higher risk and lower reliability. Applying DevOps principles and practices to include change management, security and compliance improves flow, reliability, performance, and keeps the auditors off your back.

DevSecOps transformation

Whilst some call this DevSecOps, many would simply call it DevOps the way it was always intended to be.

4. DevOps and Digital Transformation must be delivered from the bottom-up, and empowered from the top-down.

Culture is the reflection of what we do, the behaviours we manifest, the practices we perform, the way we interact and what we believe. Culture change is never successfully implemented only from the top-down, and must be driven and engaged with by those expected to actually change their behaviours and practices.

DevOps transformation promotion

Cultural barriers to change include unclear responsibilities (enter Team Topologies), insufficient feedback loops, fear of change and a low prioritisation for fast flow, and most importantly, a lack of psychological safety.

Psychological safety and risk

A lot of these findings, unsurprisingly, echo the findings from Google’s 2013 Project Aristotle, which showed that psychological safety, clarity, dependability, meaning and impact were crucial for high performance in teams.

Extra note on “Legacy” workloads.

The report highlighted the “dragging” effect that legacy workloads can have on flow and change rate, as an effect of their architecture, codebase, or infrastructure, or the fact that nobody in the organisation understands it any longer. Rather than leave alone your legacy workloads, invest in them “so that they’re no longer an inhibitor of progress”. This could be as simple as virtualisation of physical hardware, or decomposing part of the system and moving certain components to cloud-native platforms such as Kubernetes or OpenShift. Even if you have to do something a bit “ugly” such as creating 18GB containers, it’s still a step forward.

TL;DR

  1. Organisational dynamics must be considered crucial to transformation.
  2. Cloud-native approaches are critical. It is no good to simply move traditional workloads to the cloud.
  3. Shift security, compliance and change governance left, and include security stakeholders in all stages of value delivery.
  4. Culture change is key, and must be promoted from the very “top” as well as delivered from the “bottom”. Psychological safety is at the core of digital and cultural transformations.

If you’re interested in finding out more about DevOps and Digital Transformations, Psychological Safety, or Cloud Native approaches, please get in touch.

Thanks to Nigel Kersten, Kate McCarthy, Michael Stahnke and Caitlyn O’Connell for working on the 2021 State of DevOps Report and providing us with these insights.

View the 2021 Accelerate State of DevOps Report summary here.