DevOps has become common within many companies across a number of industries as design and development teams focus on creating products as fast as possible by leveraging automation, agile and DevOps processes. Still, even if you pioneered DevOps in your organization, if you don’t continuously refine and enhance your processes and practices, you risk falling behind your competitors. The business world is dynamic, and your DevOps has to be equally dynamic.
DevOps is a loosely defined term, and if you ask around you might find that most of your colleagues will say they are already following DevOps. Perhaps they are right, and the organization does have a DevOps culture and follows some or even most of the practices, processes and tools. But how do you gauge is the process is mature enough to meet current business objectives? Without systematically assessing your performance and rate of improvement, you don’t know.
If your organization already uses parts of the DevOps process-which is likely-it is important to regularly review and rebuild the business case to meet current business objectives, goals, customer demands and any specific challenges your organization is trying to address.
A self-assessment will help the DevOps team determine its current level of maturity, so it can focus attention on what needs to be done to reach the desired DevOps maturity level. Of course, there are many ways to assess the maturity of your DevOps process. But first you must understand the three overlapping dimensions of DevOps-people, process and technology-so when you perform your DevOps assessment, you cover all the bases while keeping security and compliance in mind.
Many people think that investment in DevOps is primarily about investing in tools, technologies and automation. But investing in people and process is just as important. This point is stressed in the 2016 State of DevOps Report.
The list of questions below, covers the three dimensions and is divided into five categories: business objectives and goals, people and culture, processes and techniques, automation, and technology. The set of parameters and how you answer the questionnaire depends entirely on your business objectives and goals. Pick and choose the questions that make the most sense for your business. There are many online assessment tools available, but most don’t have the flexibility to customize the questionnaire to set relative weights to match your company.
The Multidimensional DevOps Questionnaire
Assess the Current Capabilities Against Business Objectives and Goals:
• How quickly does your customer expect a response to their feedback and how long would it take for your organization to respond?
• How long would it take for your organization to deploy a single line of code change?
• Does your organization have the capability to deploy changes at a rapid pace on a repeatable and reliable basis, regardless of the skill differences among the members of the team?
• How many changes can be handled by your organization at a time? What are the throughput requirements and what is the current throughput?
• Can your organization improve the system by proactively observing user behavior and interactions and optimize accordingly?
• Can your organization scale dynamically to handle the required user traffic; either scale up and down?
• What are the competitive advantages your organization expects to achieve if DevOps is implemented to an ideal state?
• Can the organization deploy multiple versions of the same software to a different group of users and assess the feedback; for example, with A|B testing?
• How long would it take to recover the service in case of failures? Are there issues in the recovery process or any collaboration gaps?
• How does your organization measure the fitness of the product for the user needs based on real usage?
• Does your organization have up-to-date monitoring and service management practices in place?
• How are issues in production discovered and communicated to developers and other stakeholders? Is the system instrumented for proactive alerting?
• How is feedback collected from users?
• Does your organization see value in DevOps, are you thinking of justifying the investment or thinking about how to quantify the benefits?
• What additional investment is required and what is the ROI?
Assess the Alignment of People and Culture:
• Does everyone understand why the organization is doing DevOps and how it will benefit them?
• Are there any cultural issues in meeting the business objectives? For example, does your organizational structure support cross-functional team collaboration-reducing friction and barriers-with the emphasis on releasing new capabilities rather than supporting individual roles? Is there any conflict with goals or priorities between the groups?
• Are there any skillset issues? For example, is the team proficient enough to handle unforeseen issues? Does everyone release and deploy reliably? Is there a culture of sharing the learnings? Is the team skilled in the tools and technologies used for automating different stages of the delivery pipeline?
• Do all the members of the team have visibility into the work of other team members?
• Does your organization have the culture of involving operations early in the lifecycle?
• How do you measure the success of DevOps? On what basis are employees rewarded?
Assess the Current Processes and Techniques:
• Does your process support collaboration between the development and operations teams by eliminating barriers?
• Is there visibility across the end-to-end delivery pipeline? If so, has the team done the analysis and assessed the level of visibility at every step?
• What is the change management practice? Is it hindering the release of new capabilities into production at the pace required? Does the practice have the capability to trace the events between the change requests or defects and the associated code or requirements?
• Does the process support continuous improvement based on learnings and a formal feedback loop?
• Do the current metrics and measurements used across teams align to the overarching business goals and objectives, rather than to individual priorities?
• How does the organization presently measure code quality?
Assess Automation Maturity:
• Is the organization able to create production-like infrastructure quickly?
• How much of the repetitive processes are automated?
• Can the organization scale the system quickly and automatically based on user demands? If so, how quickly?
• How easy is it to reproduce a production issue in the lower environment with a similar test scenario and similar test data?
• Regarding continuous testing, is the code-quality check, unit testing, functional testing and non-functional testing automated to production-grade scenarios?
• How do you trace between requirements and test cases to ensure coverage?
• Regarding continuous integration and continuous delivery: Have the build, release and deploy processes been fully automated across all environments to meet business agility requirements?
• Do all stakeholders have visibility into what is going into production and what is happening in production? Is visual management automated?
• In the case of any unforeseen issues, can the organization automatically rollback software to the previous state without much business disruption?
• Continuous monitoring: - Is there a continuous, automated monitoring capability of the infrastructure, applications and services? Does the monitoring show the performance differences correlating to the event of change?
• What are the additional automation opportunities in the delivery pipeline?
Assess the Technology Alignment:
• Are all the services, including infrastructure and application functions, virtualized to support varying demand?
• Is the infrastructure coded and versioned?
• Does the organization leverage private or public cloud?
• Does the organization have the tools and technologies necessary to set up delivery pipeline?
• Is every stage in the delivery pipeline automated?
• Does the organization have the tools and technologies to enable automated deployment and release in place?
• Does the organization have the ability to support A|B Testing and Blue|Green Deployment?
By answering the questions above-keeping business objectives and your organization’s goals in mind-the product team can see where the strengths, weaknesses and gaps reside. Based on the self-assessment, the team can determine whether the DevOps maturity level classifies as high, medium or low.
The figure below, sourced from the 2016 State of DevOps report, provides guidance on the meaning of these three classifications.
Gauging Performance: High, Medium and Low
Source: State of DevOps report 2016, Puppet
The assessment is an important tool and should be conducted as a routine ‘health check-up’ for your performance. It’s an important tool for keeping the organization aware of where it stands in terms of DevOps maturity and where there’s work to be done. Refinements can be made based on the results of the assessment, which will enhance your DevOps performance and potentially put the company ahead of the competition regarding quality and cycle-time of product and service development.
DevOps Self-Assessment: https://devopsassessment.azurewebsites.net/
DevOps assessment: https://www.ca.com/media/cae/html/page.html