Long read

DevOps is the offspring of agile software development – born from the need to keep up with the increased software velocity and throughput agile methods have achieved. Advancements in agile culture and methods over the last decade exposed the need for a more holistic approach to the end-to-end software delivery lifecycle. 

What is Agile Software Development?

Agile Development is an umbrella term for several iterative and incremental software development methodologies. The most popular agile methodologies include Scrum, Kanban, Scaled Agile Frame work (SAF), Lean Development and Extreme Programming (XP).

While each of the agile methodologies is unique in its specific approach, they all share a common vision and core values (see the Agile Manifesto). They all fundamentally incorporate iteration and the continuous feedback that it provides to successively refine and deliver a software system. They all involve continuous planning, continuous testing, continuous integration, and other forms of continuous evolution of both the project and the software.

What Are the Challenges DevOps?

Prior to DevOps application development, teams were in charge of gathering business requirements for a software program and writing code. Then a separate QA team tests the program in an isolated development environment, if requirements were met, and releases the code for operations to deploy. The deployment teams are further fragmented into siloes groups like networking and database. Each time a software program is “thrown over the wall” to an independent team it adds bottlenecks. The problem with this paradigm is that when the teams work separately:

• Dev is often unaware of QA and Ops roadblocks that prevent the program from working as anticipated.
• QA and Ops are typically working across many features and have little context of the business purpose and value of the software.
• Each group has opposing goals that can lead to inefficiency and finger-pointing when something goes wrong.

DevOps addresses these challenges by establishing collaborative cross-functional teams that share responsibility for maintaining the system that runs the software and preparing the software to run on that system with increased quality feedback and automation issues.

What Is the Goal of DevOps?

Improve collaboration between all stakeholders from planning through delivery and automation of the delivery process in order to:

• Improve deployment frequency
• Achieve faster time to market
• Lower failure rate of new releases
• Shorten lead time between fixes
• Improve mean time to recovery

A Common Pre-DevOps Scenario The software team meets prior to starting a new software project. The team includes developers, testers, operations and support professionals. This team plans how to create working software that is ready for deployment.

Where Are You on the DevOps?

The DevOps continuum is a helpful way to look at the different aspects of DevOps. The bottom horizontal axis represents what people perceive DevOps to fundamentally be focused on. Some people adamantly feel that DevOps should focus on culture more than tools, while on the other people tend to value tools over culture.

The vertical axis depicts the three levels of the DevOps delivery chain: continuous integration, continuous delivery and continuous deployment. The DevOps community refers to organizations in the top right of the DevOps continuum as pink unicorns because there are currently so few of them that you don’t see them in the wild very often. Popular examples of these unicorns are companies like Netflix, Etsy, Amazon, Pinterest, Flicker, IMVU and Google. In a recent poll participants indicated where their organizations fit on the DevOps continuum:

Thought leaders, coaches and bloggers often portray a vision of DevOps in the upper-right corner and they will often have a strong bias towards either DevOps culture or automation tools. While it is ok to have esoteric debates about whether DevOps culture or tools are more important, the reality is that you can’t have DevOps without tools and all the tools in the world won’t help if you don’t have a strong supporting culture.

DevOps can be a blend of culture, tools and maturity that make sense for your organization and what makes sense will most likely evolve over time. The important thing is to continually strive to break down the walls and bottlenecks between the phases of software delivery by improving collaboration and automation. In the following sections, we dive deeper into each aspect of the DevOps continuum to help you better understand where you fit.

What Are the Phases of DevOps?

There are several phases to DevOps maturity; here are a few of the key phases you need to know.

Waterfall Development

Before continuous integration, development teams would write a bunch of code for three to four months. Then those teams would merge their code in order to release it. The different versions of code would be so different and have so many changes that the actual integration step could take months. This process was very unproductive.

Continuous Integration

Continuous integration is the practice of quickly integrating newly developed code with the main body of code that is to be released. Continuous integration saves a lot of time when the team is ready to release the code.

DevOps didn’t come up with this term. Continuous integration is an agile engineering practice originating from the Extreme Programming methodology. The terms been around for a while, but DevOps has adopted this term because automation is required to successfully execute continuous integration. Continuous integration is often the first step down the path toward DevOps maturity.

The continuous integration process from a DevOps perspective involves checking your code in, compiling it into usable (often binary executable) code and running some basic validation testing.

Continuous Delivery

Continuous delivery is an extension of continuous integration. It sits on top of continuous integration. When executing continuous delivery, you add additional automation and testing so that you don’t just merge the code with the main code line frequently, but you get the code nearly ready to deploy with almost no human intervention. It’s the practice of having the code base continuously in a ready-to-deploy state.

Continuous Deployment

Continuous deployment, not to be confused with continuous delivery [DevOps nirvana], is the most advanced evolution of continuous delivery. It’s the practice of deploying all the way into production without any human intervention.

Teams that utilize continuous delivery don’t deploy untested code; instead, newly created code runs through automated testing before it gets pushed out to production. The code release typically only goes to a small percentage of users and there’s an automated feedback loop that monitors quality and usage before the code is propagated further.

There are a very small number of companies that are truly practicing continuous deployment. Netflix, Etsy, Amazon, Pinterest, Flicker, IMVU and Google are popular examples of companies doing continuous deployment.

While DevOps nirvana is often not the end goal for most enterprises, they often focus on moving towards continuous delivery.

What Are the Values of DevOps?

DevOps focuses heavily on establishing a collaborative culture and improving efficiency through automation with DevOps tools. While some organizations and people tend to value one more than the other, the reality is it takes a combination of both culture and tools to be successful. Here’s what you need to know about these two DevOps values.

DevOps Culture

DevOps culture is characterized by increased collaboration, decreasing silos, shared responsibility, autonomous teams, improving quality, valuing feedback and increasing automation. Many of the DevOps values are agile values as DevOps is an extension of agile.

Agile methods are a more holistic way of delivering software. Agile development teams measure progress in terms of working software. Product owners, developers, testers and UX people work closely together with the same goals.

DevOps is just adding the operations’ mindset and maybe a team member with some of those responsibilities into the agile team. Whereas before DevOps progress is measured in terms of working software, with DevOps progress is measured in terms of working software in the customer’s hands.

To achieve this, Dev and Ops must break down the silos and collaborate with one another, share responsibility for maintaining the system that runs the software, and prepare the software to run on the system with increased quality feedback and delivery automation.

DevOps Tools

DevOps tools consist of configuration management, test and build systems, application deployment, version control and monitoring tools. Continuous integration, continuous delivery and continuous deployment require different tools. While all three practices can use the same tools, you will need more tools as you progress through the delivery chain.

What Tools Are Used in DevOps?

Earlier we briefly discussed some of the tools used in DevOps; here are some of the key tools and practices you need to know.

Source Code Repository

A source code repository is a place where developers check-in and change code. The source code repository manages the various versions of code that are checked in, so developers don’t write over each other’s work.

Source control has probably been around for forty years, but it’s a major component of continuous integration. Popular source code repository tools are Git, Subversion, Cloudforce, Bitbucket and TFS.

Build Server

The build server is an automation tool that compiles the code in the source code repository into executable code base. Popular tools are Jenkins, SonarQube and Artifactory.

Configuration Management

Configuration management defines the configuration of a server or an environment. Popular configuration management tools are Puppet and Chef.

Virtual Infrastructure

Amazon Web Services and Microsoft Azure are examples of virtual infrastructures. Virtual infrastructures are provided by cloud vendors that sell infrastructure or platform as a service (PaaS). These infrastructures have APIs to allow you to programmatically create new machines with configuration management tools such as Puppet and Chef.

There are also private clouds. For example, VMware has vCloud. Private virtual infrastructures enable you to run a cloud on top of the hardware in your data center.

Virtual infrastructures combined with automation tools to empower organizations practicing DevOps with the ability to configure a server without any fingers on the keyboard. If you want to test your brand-new code, you can automatically send it to your cloud infrastructure, build the environment and then run all of the tests without human intervention.

Test Automation

Test automation has been around for a long time. DevOps testing focuses on automated testing within your build pipeline to ensure that by the time that you have a deployable build, you are confident it is ready to be deployed. You can’t get to the point of continuous delivery where you’re fairly confident without any human intervention that your code is deployable without an extensive automated testing strategy. Popular tools are Selenium and Water.

Pipeline Orchestration

A pipeline is like a manufacturing assembly line that happens from the time a developer says, “I think I’m done,” all the way to the time that the code gets deployed in the production or a late-stage pre-production environment.

Unifying Enterprise Software Development and Delivery

Version One VS unifies agile application lifecycle management and DevOps, providing a full picture of your entire software delivery pipeline in a single platform. Version One Continuum for DevOps is an enterprise continuous delivery solution for automating, orchestrating, and visualizing the flow of change throughout the software delivery cycle

Best Practice

Understand the collaboration and shared tools strategy for the Dev, QA, and infrastructure automation teams

DevOps teams need to come up with a common tools strategy that lets them collaborate across development, testing, and deployment (see Figure 1). This does not mean that you should spend days arguing about tooling; it means you work on a common strategy that includes DevOps…

• Processes
• Communications and collaboration planning
• Continuous development tools
• Continuous integration tools
• Continuous testing tools
• Continuous deployment tools
• Continuous operations and Cloud Ops tools

Coming up with a common tools strategy does not drive tool selection — at least not at this point. It means picking a common share strategy that all can agree upon and that is reflective of your business objectives for DevOps.

The tool selection process often drives miscommunication within teams. A common DevOps tools strategy must adhere to a common set of objectives while providing seamless collaboration and integration between tools. The objective is to automate everything: Developers should be able to send new and changed software to deployment and operations without humans getting in the way of the processes.

Use tools to Identify the Plans

No ad hoc work or changes should occur outside of the DevOps process, and DevOps tooling should capture every request for new or changed software. This is different from logging the progress of software as it moves through the processes. DevOps provides the ability to automate the acceptance of change requests that come in either from the business or from other parts of the DevOps teams.

Examples include changing software to accommodate a new tax model for the business, or changing the software to accommodate a request to improve performance of the database access module.

Use tools to log metrics on both manual and automated processes

Select tools that can help you understand the productivity of your DevOps processes, both automated and manual, and to determine if they are working in your favor. You need to do two things with these tools. First, define which metrics are relevant to the DevOps processes, such as speed to deployment versus testing errors found. Second, define automated processes to correct issues without human involvement. An example would be dealing with software scaling problems automatically on cloud-based platforms.

Implement test automation and test data provisioning tooling

Test automation is more than just automated testing; it’s the ability to take code and data and run standard testing routines to ensure the quality of the code, the data, and the overall solution. With DevOps, testing must be continuous. The ability to toss code and data into the process means you need to place the code into a sandbox, assign test data to the application, and run hundreds — or thousands — of tests that, when completed, will automatically promote the code down the DevOps process, or return it back to the developers for rework.

Perform acceptance tests for each deployment tooling

Part of the testing process should define the acceptance tests that will be a part of each deployment, including levels of acceptance for the infrastructure, applications, data, and even the test suite that you’ll use. For the tool set selected, those charged with DevOps testing processes should to spend time defining the acceptance tests, and ensuring that the tests meet with the acceptance criteria selected.

These tests may be changed at any time by development or operations. And as applications evolve over time, you’ll need to bake new requirements into the software, which in turn should be tested against these new requirements. For example, you might need to test changes to compliance issues associated with protecting certain types of data, or performance issues to ensure that the enterprise meets service-level agreements.

Ensure continuous feedback between the teams to spot gaps, issues, and inefficiencies

Finally, you’ll need feedback loops to automate communication between tests that spot issues, and tests that process needs to be supported by your chosen tool. The right tool must identify the issue using either manual or automated mechanisms, and tag the issue with the artifact so the developers or operators understand what occurred, why it occurred, and where it occurred.

The tool should also help to define a chain of communications with all automated and human players in the loop. This includes an approach to correct the problem in collaboration with everyone on the team, a consensus as to what type of resolution you should apply, and a list of any additional code or technology required. Then comes the push to production, where the tool should help you define tracking to report whether the resolution made it through automated testing, automated deployment, and automated operations.

Pick up the Automation Tools to execute the flow of work deliver for the project driven based

The major tool categories include: 

• Version control: Tools that track software versions as they are released, whether manually or automatically. This means numbering versions, as well as tracking the configuration and any environmental dependencies that are present, such as the type, brand, and version of the database; the operating system details; and even the type of physical or virtual server that’s needed. This category is related to change management tools.
• Build and deploy: Tools that automate the building and deployment of software throughout the DevOps process, including continuous development and continuous integration.
• Functional and non-functional testing: Tools that provide automated testing, including best practices listed above. Testing tools should provide integrated unit, performance, and security testing services. The objective should be end-to-end automation.
• Provisioning and change management: Tools to provision the platforms needed for deployment of the software, as well as monitor and log any changes occurring to the configuration, the data, or the software. These tools ensure that you can get the system back to a stable state, no matter what occurs.

Written by Senthilkumar Sivakumar, cloud architect and DevOps specialist