Pavan Keshavamurthy

Pavan Keshavamurthy

Recent Posts

Why you need a DevOps Audit

Posted by Pavan Keshavamurthy on Dec 21, 2017 3:19:00 PM

You have convinced your team to start adopting DevOps practices. They have automated some processes, and are using tools like Jenkins and Travis.CI for continuous integration. You see the team working more efficiently than before, but you are yet to see the promised “significant acceleration in the delivery pipeline”.

Sounds familiar?

That could be because your team has gone the DevOps way, but not in the right manner. According to Mike Kavis, VP, Cloud Technology Partners, here’s what could be happening:

  • A New Silo: You hire a new set of people, the DevOps engineers, whose job is to take care of all the automation and work with the new tools. But that’s just adding another silo, and not really solving anything.

  • No Ops Needed: The Dev team decides to take up the infrastructure-as-code principle, and provision their own environments. Their work gets faster, but the challenges in network, security compliance, and support, still remain.

  • Rebranding Ops: The Ops team gets rebranded as a DevOps team. They’ve solved the problems with provisioning and deployment, but the Dev team’s challenges around configuration management, testing, and continuous integration still remain.

 

Your DevOps practice could have fallen into any, or all of these traps. 

How do you get from here to a stage where DevOps is bringing in measurable ROI?

You need to identify where you stand on the DevOps maturity model, and how to move forward. And that’s when you required a DevOps audit.

What to Expect Out of Your DevOps Audit

All enterprises, in their quest for adopting DevOps, go through the DevOps maturity model

Why You Need a DevOps Audit

They usually move from being a siloed organization to slowly adopting different DevOps practices, starting with automation. And finally, they become a DevOps practitioner with one-touch deployment, faster bug resolutions, and reduced system failures.

At the initial stages, the team itself is concentrating on collaborating with each other, and integrating automation. But there’s no bird’s eye view of where their DevOps practice is headed. 

A DevOps audit at this stage helps enterprises:

Benchmark their DevOps efforts against industry best practices

This involves identifying your position on the DevOps maturity model and assigning a DevOps maturity score for your enterprise. It’s helpful for the team if this score is presented as a breakdown of different aspects of the delivery cycle. For example: individual scores for performance/processes in terms of version control, deploy automation, lead times, failure notification etc.

Understand the current bottlenecks in their delivery pipeline

The audit should identify sections of your pipeline where you require better processes, or those that could cost you a successful roll-out. It should also present actionable insights that tell your team:

  • which processes to automate

  • what are the right tools for the job

  • what are the flaws in their current risk management and rollback strategies

 

The idea is to give your team a set of immediate next steps to work upon.

Identify low hanging fruit

You should expect your audit to pinpoint sections of the pipeline where DevOps practices can generate the highest initial impact. 

For example: If the QA team never gets adequate time to test each build, leading to increased bug reports, the audit should showcase that as your low-hanging fruit. 

In this case, you can have an automated script that will perform a standard set of tests for each commit, before pushing the code forward. That will significantly speed up the development, while ensuring lower errors. 

Chart out a Roadmap 

The audit should lay out a plan for your teams to advance on the DevOps maturity scale. You should expect a DevOps playbook that lays out the right practices and toolchains, to make your teams competent and competitive.

For enterprises to retain their competitive edge against industry disruptors, DevOps maturity is critical. Teams with a high-performing DevOps methodology can deploy code 46X more frequently, and have a 96X faster mean time to recovery. Those are hard to ignore gains.

So, are you ready to audit your DevOps practice? Drop us a line and explore how our DevOps consulting team can help.

Topics: CI/CD & DevOps, Framework and Libraries

How to chalk out your Enterprise DevOps adoption strategy

Posted by Pavan Keshavamurthy on Dec 15, 2017 5:45:00 PM

There are a lot of steps from product build to deployment, and also a lot of things that can, and do, go wrong. A few of these might sound familiar:

  • The Dev team adds a new feature right before release and the QA team does not have enough time to run the whole gamut of tests. The code is pushed to live and it’s too late before you realize that you missed a few bugs.

  • You update your product and release a new version. It worked fine for the developers and testers, but not when deployed on the production servers. Because the Dev team missed informing the Ops team about updating a library or the database on the servers.

  • Your code is ready to go, but the Ops team says they will need a couple of days to configure all the environments to the given specifications. Or maybe a server goes down, and they take hours to get it back up, manually configuring it all over again.

 

Every product team has faced these challenges at some point in time. If your product development and delivery pipeline is largely manual, the only way to avoid these challenges is to work steady and double check to make sure you haven’t missed anything.

Meanwhile, that new disruptive start-up in your industry has already released the second version of a competing product. They are able to do this because of shorter and faster release cycles, ability to deploy quick fixes for bugs, and recover faster from system failures.

And this is probably why your enterprise has to start looking towards DevOps adoption. If you wish to stay ahead of the curve with your products and services, this is no longer a matter of choice. Besides resolving some of the current challenges your siloed teams face, DevOps is key to a faster product delivery pipeline.

However, enterprise-wide DevOps adoption is easier said than done. There is, of course, the initial resistance to changing established practices. Additionally, convincingly showcasing the value of DevOps to the entire organization is a challenge.

Micro Hering, Principal Director at Accenture, points out one of the key reasons why DevOps adoption is derailed: “Some group goes off and implements DevOps practices (for example the testing center of excellence or the operations team) and they have great successes. Their individual results improve so that the automated regression now runs in minutes not hours and giving developers a development environment from the cloud takes hours not weeks. Yet the organization as a whole fails to see the benefits because the different practices are not compatible or too many dependencies continue to hinder real results.”

Hence, what is needed is a well-planned and executed DevOps adoption strategy, that will produce measurable results. What we suggest is a two-phase roadmap:

Phase 1: Showcasing DevOps ROI
Phase 2: Identifying and Side-stepping the Trip Wires

Showcasing DevOps ROI

To accelerate time to market with DevOps, it has to be adopted by all teams across the organization. And that is easier to achieve when stakeholders get behind the idea of DevOps adoption. So the first step is to demonstrate to all stakeholders how DevOps can bring in significant benefits.

Here’s how to do that:

Evaluate Your Delivery Pipeline

Understanding the existing delivery pipeline is the first step. Get together all the process stakeholders. Map out your pipeline in complete detail, understanding each process, highlighting if it’s manual or automated, and how long it takes to complete it. Identify any inherent cause-effect relationships and dependencies in the pipeline.

Identify Process Bottlenecks

The next step is to identify all the existing process bottlenecks. 

This could be in terms of time taken by the Ops team to configure a production environment. Or the time taken by the QA team to thoroughly test every feature addition. Or mismatch in configurations between development and deployment servers. 

Identifying these bottlenecks helps you and other stakeholders realize that there is a need for adopting better practices. It also demonstrates where the distinct DevOps practices like infrastructure-as-code, automated test scripts, configuration management etc. will fit in.

This is also the right time to identify certain base performance metrics, which will help showcase the improvements made with DevOps.

Choose Your Experimental Set

A lot of times, applying DevOps principles to an isolated process is not enough to convince stakeholders of the benefits. It gets viewed as a one-off incident and not something that would be replicable across the organization.

Your experimental set, i.e. the specific set of processes on which you want to apply DevOps practices, has to be chosen carefully. 

Ideally, it should meet the following criteria:

  • The processes are integrated and need to change together in order to work

  • They are important for the enterprise and whose improvement will deliver significant benefits

  • Can be optimized within a short period of time

 

For example, let’s say you choose server configuration as your experimental set, which will involve:

  • How you configure a development, testing, or production server, and how long it takes

  • How do you update the servers when new product versions are released

  • How fast can you reconfigure a server that went down

 

This inter-related set of processes can be tackled with the DevOps practices of infrastructure-as-code and configuration management. And an improvement in your server configuration times means a huge acceleration in your delivery pipeline. 

Practise DevOps

With your experimental set of processes identified, start integrating DevOps practices from the ground up. The earlier on in the process you introduce it, the better. The key here is to be conscious of the process changes and making sure the team sticks with them. Tight feedback schedules and team’s complete alignment with the final goal would be crucial. 

As you work on optimizing these processes, the benefits of DevOps would start reflecting. In the time taken to delivery, the number of bugs reported, the automations introduced and their impact on performance. These benefits must get documented and mapped against the base metrics identified. 

Now your work is ready to be showcased to stakeholders as an example where DevOps created significant ROI for the enterprise. Once they see the measurable value additions, it’s easier for them to buy into the idea and make a strong push for DevOps adoption across the enterprise. 

Identifying and Side-stepping the Trip Wires

With key stakeholders on-board, the next phase is to make sure you foresee the challenges that can come up, and how to get over those. This involves taking a look at three key factors:

Organizational Practices

DevOps demands close collaboration between Dev and Ops teams and a work culture that is focussed on accepting and correcting mistakes rather than pointing fingers. This kind of cross-team cooperation could be difficult to get right at the first go. But recognizing the need for it, and enabling it across the organization is the first step here.

Enterprise teams should also consciously extend the development methodologies of agile, to the operations teams. This allows both teams to communicate in stand-ups and retros, building a greater understanding and collaboration.

Legacy Modernization

Here’s where most enterprises trip up because they feel:

  • DevOps practices would only be effective with modern applications and platforms

  • Legacy modernization would mean completely doing away with the old architecture, which will involve significant expenditure

 

However, DevOps practices can work as easily with legacy platforms as with modern ones. Continuous integration, one-touch deployments, agile release cycles are all possible on legacy platforms, with the right tools.

Moreover, legacy modernization does not always mean an expensive upheaval. In most cases, it can be achieved through adapting your heritage systems to modern methods. And DevOps practices can actually let you do this faster and in a more efficient manner.

System Reliability

The speed of delivery achieved through DevOps often raises concerns around system reliability.

As a concept, DevOps embraces failure as inevitable and concentrates on designing systems that can get back to work fast, after a breakdown. There are examples like Netflix’s Chaos Monkey, that are designed to trigger randomized system failures. This pushes their teams to build systems that are capable of healing fast in such conditions. 

So DevOps moves beyond reliability, towards achieving more resilient systems.

While this two-phase roadmap is a great starting point as enterprises start thinking about DevOps, you will have to tweak it suit your particular organization. 

There are, of course, a lot of decisions to be made once you start down the DevOps way. The most important of those is choosing the right DevOps toolchain for your teams. This is also where  DevOps consulting services like ours could lend a hand. 

Let's start the conversation about how we can power your competitive advantage with DevOps.

Topics: CI/CD & DevOps, Enterprises

How DevOps solves the challenges faced by siloed teams

Posted by Pavan Keshavamurthy on Dec 5, 2017 12:27:00 AM

The most critical factor that determines the quality of your product or service is how your Dev and Ops teams work together. If your Dev team is responsible just for production, and Ops team is concerned only with maintaining availability, the product is in trouble.

DevOps integrates these two teams, making them jointly responsible for delivering a product to your customers’ satisfaction. Adopting the core DevOps principles, you achieve:

  • Robust, quality software, built and maintained using the best practices
  • Faster time to market with smaller but frequent releases
  • Proactive monitoring that identifies bugs and issues before they become too well-baked into the product

 

Besides these key business benefits, DevOps principles also solve certain common challenges that siloed teams face:

Challenge 1: Whose fault is it?

With siloed Dev and Ops teams, every time something breaks, the first question is “Whose fault is it?” With skill-based teams like this, people tend to believe their skills are above reproach, and hence the problem must be with the other team. 

Solution: Culture

With DevOps, teams adopt a culture that’s open, trusting, and collaborative. This is best achieved by switching to project-based teams working towards a single goal.

Project-based teams make sure that the entire team, whether they are developing the product or maintaining it, are communicating with each other. The Ops people know how the Devs are working, their issues, and timelines. The devs understand the kind of challenges that Ops face, making for easier release management. And when something breaks, the team’s first priority is fixing the project, not pointing fingers.


Challenge 2: The Genius Bottleneck

All teams have that one person who takes the initiative to fix a problem whenever they see one. Maybe someone on your dev team is tired of asking the Ops team for new environments. They write scripts to quickly set up the virtual environments that they want. 

It’s an efficient solution but not a scalable one. Because only the one who wrote the scripts can troubleshoot them, and the team is stuck when that person is not around.

Solution: Automation

With DevOps, automation becomes a key characteristic of how a team works. The entire team adopts toolchains that ensure standardized automation for repetitive tasks. And each team member is capable of understanding and troubleshooting these automated processes.

Two of most common automations adopted are:

Continuous Delivery: Every piece of code is run through a series of automated tests, and error-free code is packaged and deployed. That is why teams adopting DevOps have smaller releases, where it’s easier to identify and fix bugs.

Infrastructure as code: Provisioning becomes as simple as a modular code that can be run multiple times to set up identical instances. Any change in configuration, or even reprovisioning an instance becomes a quick and reliable process, making life easier for the Ops team.


Challenge 3: Unplanned Work and Dealing with Failures

For any product or service, your systems are bound to break down at some point. That means a lot of firefighting and unplanned work for both the Dev and Ops teams. And while they are busy dealing with such immediate problems, there’s no time to think about developing strategies and practices that help anticipate and prepare for system failures.

Solution: Lean and Agile

While your development team already follows the agile and lean way of working, DevOps extends those practices to your Ops team as well. 

For the Ops team, being lean would mean learning to accept system failures as inevitable and prepare for it. Every emergency is not just a mad scramble to get things fixed, but also a close a collaboration between the Dev and the Ops team to improve the system to prevent a repeat occurrence.

Additionally, with project-based teams, the Ops team is now also a part of the regular stand-ups and retros by the dev team. So they are always in the loop about the kind of work they can expect and plan accordingly.

If you aim to build great products and services, DevOps is non-negotiable. Srijan's DevOps consulting teams can help your teams assess current practices and identify the correct DevOps adoption roadmap. Let's start the conversation about how we can power your competitive advantage with DevOps.

Topics: Project Management, CI/CD & DevOps

The key elements of a High-Performing DevOps methodology

Posted by Pavan Keshavamurthy on Sep 4, 2017 3:45:00 PM

What should an enterprise do to improve the performance of their DevOps practice? More, or less communication? Remove policies that keep the Dev and Ops teams in silos? More, or less automation?

There are a hundred different ways of looking at this. And we might still not have clarity on what one can do to have a higher performance DevOps methodology. But here’s something that can give us clues: 2017 State of DevOps Report by Puppet and DORA (DevOps Research and Assessment).

The report, now in its sixth year, received responses from 3200 people in the domain around the world. It “looks at the statistical relationships between IT performance, organizational performance, technical practices, cultural norms, and management”. While transformative leadership has emerged as key for DevOps success in this version of the report, we will keep our focus on the technical aspects that the report highlights.

Let’s look at what the report has identified as key parameters that drive high performers in the DevOps space. But first, let’s look at how high performers do better than low performers.

 

The survey found that in comparison to the low performing DevOps teams the high performers have:

  • 46X more frequent code deployments
  • 440X faster lead time from commit to deploy
  • 96X faster mean time to recover from downtime
  • 5X lower change failure rate

 

What does that translate to? The table from the report illustrates:

Devops methodology survey questions

Given the metrics we see above, there is a lot to be learned from the high-performance DevOps teams and their DevOps methodology. What does the report suggest?

Elements of a High-Performance DevOps Methodology

Loosely coupled architecture to practice continuous delivery

If the team is replacing or modifying a component, or service, does it require them to make corresponding changes to the services or components dependent on it? If yes, then that’s a bottleneck for you to consider.  For example, do minor changes to an application cause breakage? Or is the continuous integration/continuous deployment process crippled by deployments that are big and time taking? These are some warning alerts that could make you reassess your architecture. A couple of approaches for this include:

  • Use of APIs, bounded contexts to decouple large domains
  • Use of test doubles and virtualization to test services or components in isolation

 

Similarly, can delivery teams test, deploy and change their systems without depending on other teams? This could be for additional work, resources, or approvals. And can this be done with less back-and-forth communication? Do they need to check schedules with many people, just to get their job moving? The report found that high performing teams have less dependency on other teams thereby speeding up release cycles.

Daily code merging in trunk based development

Periodically merging code to trunk is part of the development workflow for most teams. However, what should be the periodicity or frequency of the merging? To get higher software delivery performance, merge code into trunk on a daily basis, have branches or forks with lifetimes less than a day, and don’t have more than three active branches at a time.

That may sound quite the contrasting approach to software development workflows being followed, but it works. 

The survey found that teams that do not have code lock periods have higher software delivery performance. Developers in high-performing teams work in small batches and develop off of trunk or master rather than long-live branches, says the report. High performers have branch life integration lasting just a few hours, while low performers have it lasting days, typically. 

More automation

The report says that high performers are doing significantly less manual work than low performers, and so have automated:

  • 33 percent more of their configuration management.
  • 27 percent more of their testing.
  • 30 percent more of their deployments.
  • 27 percent more of their change approval processes.

 

With more automation comes more time for the team to do value-creation activities, building out new features. The report cites that “by undertaking a continuous improvement initiative and investing in automation — including significant investments in automated testing — HP LaserJet was able to increase time spent on developing new features by 700 percent.”

What we've also learned, in our work with teams and in conducting research, is that people are not very good at estimating how automated they are. What they are good at is estimating the percentage of their work that is still done manually. That's not surprising: Manual work is painful, so people are highly aware of it. Once work is automated, it's no longer painful and tends to disappear from people's attention.

percentage of work done manually

The table shows high IT performers report the lowest amount of manual work across all practices — and therefore, the highest amount of automation.  

While medium performers do more manual work than low performers when it comes to deployment and change approval processes, the authors regard this phenomenon of manual work and rework as a temporary stage in which the organizations add more manual controls around changes, inevitably slowing them down. The recommendation is not to give in to the temptation and move change review process to earlier on in the development cycle.

Quality and security

It is one thing to be able to deploy code rapidly, and on demand. But what of quality and security? The report measures unplanned work and rework as proxies to measure quality, and found that high-performing organizations spend 21 percent less time on unplanned work and rework. So they can spend 44 percent more time on new work, such as new features or code. They also spend 50 percent less time remediating security issues than low performers. These results point to the fact that dev teams need to involve security and quality teams in the development process early.

These are some of the key technical intervention areas that can lead to a higher-performing DevOps team.

What are some of the issues your teams face in their DevOps processes? Drop a mail here to brainstorm about your challenges and explore how our DevOps consulting services can help you resolve them. 

Topics: CI/CD & DevOps, Architecture

Discussion

Write to us

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms Of Service apply. By submitting this form, you agree to our Privacy Policy.

See how our uniquely collaborative work style, can help you redesign your business.

Contact us