As DevOps becomes the norm, more and more companies are adopting this way of working. Telecommunications firm O2/ Telefónica is no different to this. In fact, it means so much to them that they brought in Plutora, a value stream management organisation, to help centralise test environment management. Doing this enabled them to create a new environment in which they could effectively increase agility and resiliency. Jeff Keyes, Director of Product Marketing at Plutora spoke about what the implementation means for telecommunication.

Tell us about your work with O₂/Telefónica?

Plutora was brought in alongside NTT Data, which manages several test environments, alongside other hardware and consultancies managing independent systems and environments. NTT sought to manage all requests and environment usage and utilized Plutora in that effort to centralise all environments (those from NTT and those from others).

O₂ initially struggled to manage its test environments as any number of actions resulted in downtime for the environment. These are issues such as multiple teams deploying conflicting changes to the same environment, environment contention (not knowing someone else was using it), misconfigurations, and a lack of maintenance to the environments made them fragile. Test environment outages reduced the test team’s time and ability to verify features and ensure quality delivery. Meanwhile, much of the environment management team’s time was spent merely managing the schedules, environment requests, and configurations, rather than being able to focus on provisioning, automation, and maintenance.

How has Plutora brought a positive change?

Using a system to manage environments (rather than home-grown tools or spreadsheets) created a real-time connected view into what was actually going on. The benefits that O₂ has seen include:

1. Reduction in meetings and reliance on spreadsheets. No longer does the environment management team spend time managing spreadsheets, responding to phone calls and email requests (of which they had around 30,000 per year), and doing repetitive administrative tasks. The tooling enabling the team to focus on managing the workload, using data to focus improvement efforts, and improve quality service with the same staff.
2. Application delivery improved. Downtime for the environment has resulted in less time to test, rather than delayed schedules. Overall, bugs are found sooner and fixed more quickly.
3. The team is now able to consolidate usage of environments and focus on larger initiatives such as migration to cloud and improvements in development methodologies.
4. O₂ is moving towards a completely self-service model enabled by the environment management team, automation and tools. Provisioning and conflict resolution are being included in that effort.

 What other big things are happening in the world of telecommunications?

The world of IoT is exploding in every industry, including telecommunications. While testing internal software delivery against simulators is critical, it still doesn’t substitute for actual device compatibility and functionality testing on real devices. The environment labs are another type of test environment that now must also be managed. That represents a problem for most companies that don’t have centralised environment management, as similar problems arise in the form of scheduling conflicts, misconfigurations, and downtime causing overall project delays.

 What problems tend to arise in a telecommunications and testing environment and how do you overcome these?

Telecommunications companies have typically been around for a very long time. This means they have a long legacy of software and hardware products forming a fabric of solutions for their customers. Every application built must be compatible with the legacy systems (both hardware and software). In addition to that complexity, telecom is highly regulated meaning compliance adds time, process, and extra steps to each effort. As test environments look to mimic production, the complexity required for non-production is very high spanning architectures, methodologies, geographies, vendors and products. Each team must ensure alignment of their efforts to conform in an auditable way.

To overcome these problems, teams simply can’t deal with test environments in an ad-hoc way. Without automation, teams will never get on top of the requests for environments, let alone be able to improve the process of managing them.

You have written about release management; do you think this is something still relevant today? Why?

The job of release management focuses on managing the resources and risks of all application release trains. Release management also adds governance to ensure regulatory compliance of each pipeline regardless of the technology utilised. In product-oriented, DevOps style teams, there has been a push that release management is antiquated and can simply be automated.

As DevOps teams mature, they typically transform their relationship with release management from that of oversight and management to the release management coach. Release management teams are implementing more tooling to enable the increased velocity of each team and the explosion of smaller, autonomous teams. They use these tools to guide each effort for all teams regardless of their level of maturity. They commonly use the analogy of providing the “release highways” that include guardrails, down which each autonomous team can drive whatever car they’d like at whatever speed suits them.

Organisations that don’t resolve the new relationship of release management and DevOps stall in “wrestling matches” that are generally halted by the operations teams. As heard by one customer onboarding to Plutora, “We’ve been doing DevOps for 18 months – and have yet to ship anything.”

What big things do you predict for 2020?

• Now is the time of value stream management – using systems thinking to manage each application delivery pipeline independent of its tooling, architecture and mythology. Using metrics gathered as part of the process to drive further innovations to improve efficiency, quality, velocity and ultimately, value delivered to the customer.
• Autonomous teams will drive massive change in the way development is operated. Product managers will “own” not only each idea as it gets delivered into production, but also customer adoption and satisfaction. As an industry, we’ll see a continued push towards, “You built it, you run it” and no longer will teams have any walls to throw their applications over. These teams will be incredibly nimble and wrestle with how to provide governance and transparency to the business.