DevOps & Data

I’m Lee Gardiner, I’m currently the Head of Infrastructure & Service for the Collinson Group and I adore technology transformations, cloud migrations, and being a leader.

Over the years, I’ve seen many differing opinions as to “What is DevOps?” and typically when I ask the question in interviews, I’m given differing answers. In this article, I’d love to describe what DevOps is to me as well as my observations.

Part I – What is DevOps?

I’ve been fortunate to see DevOps transformations at scale as well as false starts. When I initially started looking at DevOps, it scared the life out of me – “Engineers with version control, ChatOps and being friendly with Developers, madness!” –  Developers were the enemy, they force change with no regard to our wonderful running systems that we barely monitor!

So what changed?  For me, it was seeing a business I loved close and 500 people out of a job, literally the worst-case scenario of the Phoenix Project (if you haven’t read it, please do). This was one of the worst experiences of my professional career and, in hindsight, it was already in motion well before I joined.

The realisation that my goal is fundamentally the same as a Developers put me onto a DevOps path, working closely with Developers for shared goals and my personal favorite bit of DevOps – shared wins.

In digital transformations, businesses typically upskill SysAdmins into DevOps Engineers (I’m not a fan of the title, but it’s one I’ll not fight for now), businesses throw brand new tools at engineers, send them on training courses, and Voila: DevOps Engineer, still in an infrastructure team and not really providing any major benefits to the business, we refer to this as DevOps as a function and in my opinion, it’s missing the point entirely.

DevOps to me is about that shared win, about providing velocity to the business, its processes, and its staff, and it’s why I’m a fan of the scaled agile framework. There are clear and concise ways of implementing DevOps into your organization.

Medium to long term DevOps Engineers run in a chapter, embedded in development squads, and evangelise all things DevOps while embracing shared goals and shared wins. Which is wonderful and will return immense value and lower time to markets, but what next?

Consider the abstractions that we’ve made from a SysAdmin to DevOps engineer

Network Layer – Abstracted due to Cloud

Server Hardware Layer – Abstracted due to cloud

Operating System Layer – Abstracted due to containerisation

Monitoring – Now as code, shared by all

CICD – Now as code, shared by all 

While it’s amazing to go down this journey, the leap I mentioned before means we are now closer to being a Developer than a SysAdmin! We continue to abstract these layers and make it less and less complicated to run applications in a scaled, safe, reliable fashion.

The end goal and the aim is to transition away from DevOps Engineers into Developers, and we will no doubt get there while continuing to embrace and evangelise further abstractions away from infrastructure. DevOps just becomes something we in technology do and we’ll see other specialisms absorbed into this view with the likes of QA’s and DBA’s not being too far down the track.

I hope I’ve not bored you too much with my personal idealistic view of DevOps and I welcome any and all feedback you’d like to send my way.


Part II – Data at your fingertips

When I started as a Jnr SysAdmin many moons ago, the first sign of an issue was generally an angry customer at the other side of a ticket, monitoring was done in a black box manner (basically checking if a port was open or not) and the only true monitoring was done around bandwidth consumption to ensure correct billing.

Over the years, the way we monitor systems and the amount of logging we do out of the box has changed dramatically. We now find ourselves with the reverse problem in that we’re swamped with alerts and logs.

As a leader, my role is made considerably less complicated by being data-centric, the metrics that are important to me are shared, transparent and on my part, memorised for usage throughout the week.

As an engineer, my role was made considerably easier having accurate chaos engineered data, simulating environment failures, and having predictability, typically while we may log data – are we actually putting it to work?

Data logged must be of use to someone, somewhere, otherwise, there is little to no point in storing it let alone triggering alerts or actions off it, the fight to reduce noise is not a straightforward one to win and will take true collaboration to achieve.

It is therefore vital to have well-defined guardrails or operational acceptance tests to ensure prior to a service going live we define how it should log and on what events we need to be alerted upon. Services that are live prior to these processes should be reviewed and improved as part of the application/platform lifecycle.

“Isn’t this where SRE’s come into play?” – this is a shared goal and a shared win, something we as engineers should be focussing on to improve the customer experience.

Put your data to work, be transparent on it, and be critical with it if it’s not providing value, like anything else this is another area of constant change, constant advancements, and more importantly constant learning.


Written by Lee Gardiner, Head of Infrastructure & Service at Collinson