So, it is no surprise that interpreting the implications of million-plus test execution is a challenge, even for the most skilled team.  Sure, they can – and almost certainly have – embraced automated continuous testing (CT) to a greater or lesser extent, but in many instances, that may not be sufficient. There is still a huge gap in implementing test automation that runs in a reliable, stable and efficient way throughout the DevOps pipeline.

To put some context around that, anecdotal evidence suggests that more than 40 percent of failed automation attempts were a direct result of scripting issues, thus undermining the value of CT. In other words, scripting is probably the single biggest cause of automation failure, and without automated testing, DevOps-at-scale is hard, arguably impossible, to achieve.

This is why more organisations are now looking at how using machine learning can help make sense of all the noise that test results create, and understand the real business impact, is the only way to deal with the scale of test data that modern software development lifecycles inevitably require.

Machine learning

Machine learning (ML) has become one of the most popular tech terms on the planet, something that even individuals with no involvement in the technology industry or its use are aware.   Like something out of science-fiction, it has caught the imagination of people worldwide, with both positive and negative connotations.

That aside, when it comes to the latest techniques and tools for automated testing, machine learning is beginning to make waves and, together with other elements of smart analytics, are becoming an essential part of the root cause analysis (RCA) toolbox.  On a very practical level, ML can eliminate the need to write and maintain test scripts.  Instead, changes to test scripts are made agnostically, so scripts continuously run and self-heal, all without getting in the way of other aspects of the development pipeline.

The potential value of machine learning is particularly evident in mobile and web app testing because these are very fragmented and complex platforms to handle and understand.  What ML can do in this context is to keep all those platforms visible, connected, and in a ready-state mode.  In a test lab, ML helps to surface when something is outdated, disconnected from WiFi, or another problem – and moreover, help understand why that has happened.

Another way in which ML helps is through showing trends and patterns, helping to not only visualise all that data but provide further insight and make sense of what has happened over the past weeks or months. For instance, it can identify the most problematic functional area in an application, such as the top 5 failing tests over the past 2-3 testing cycle, or which mobile/web platforms have been most error-prone over the past cycles.   Was a failure caused by the lab, was it a pop-up, or a security alert?

This really matters.  Teams invest time, resources and money in automating test activities, but where all this really has an impact and add value is at the reporting stage.  Up until that point, those tests are being executed behind the scenes and so it is hard to assess whether they are identifying real issues or not.  Test analysis at scale is when teams really understand what is happening within their software, whether those tests are of any value (and if not, why not).  Test analysis at this level of sophistication and at scale is the point at which most organizations will definitely realize the value of test automation while providing the data to identify how it can be improved.

Of course, this all ties in with – and supports – continuous integration, which is both the engine and the glue from coding to test to release to production.  As we all know, historically testing can get in the way of release deployment, so the greater the visibility into the CI, the lesser that risk.  In fact, this is how ML-based testing can actually expedite up release cycles, rather than be a roadblock, by providing that insight, reducing risking and supporting continuous improvement.

Banking example

Let’s take an example in the banking sector because there are so many unique scenarios against which to test, including smart authentication on different mobile devices, the ability to deposit a check through the camera, or find a branch through location services.  When the user accesses the camera for the first time, he or she will probably see a pop-up asking for permission to use the camera from the application.

What ML-based testing can do is discover early that the pop-ups were not handled properly, and that will save a lot of future debugging and hence possible delay in execution.  ML testing can identify during runtime that something was neglected around those pop-ups, or perhaps the security alerts connected to user permissions created an issue.

Cultural challenges

However, tools are on their own are not enough: just as important is adopting the right cultural attitude within the organization to modern test processes.  ML has had a bad rap in some quarters of the popular press for how it might take away people’s jobs in the future.  Is this true in testing?  What we are actually seeing right now from early adopters is that ML is helping test engineers evolve their careers.  Freed from writing test scripts, they can focus on more complex tasks that require human brainpower.  ML also helps ‘teach’ good testing practice, and also brings it within reach of more individuals.  Test or QA managers can expand their teams more easily, with more seasoned test professionals providing mentoring alongside being able to focus on more rewarding daily tasks, such as those tests that do not yet lend themselves to full automation.

It is early days for ML in testing and there are various levels of adoption, and maturity within the ML testing solutions offer.  Despite these variations, one thing is clear: ML-based testing is here to stay and it will continue to develop over the next couple of years until a stable level of autonomous test automation can be reached and maturity of ML achieved.  In the meantime, ML-testing can already start to have a positive impact on CT in large-scale DevOps environments, removing unnecessary resources being spent on mundane tasks, reduce risk, and accelerate release cycles.

Written by Eran Kinsbruner, Chief Evangelist at Perfecto (by Perforce)

The post Why the sheer scale of DevOps testing now needs machine learning  appeared first on DevOps Online North America.

So, how has the role changed over the last decade? And more importantly, how can developers cope with the pace of change they’re facing now? During my fifteen years in enterprise software development, I’ve witnessed much of this transformation first hand; as I helped develop products and solutions for optimising software quality and application lifecycle management – all against the backdrop of a fast-changing and unpredictable business environment.

The need for speed

The most significant change in the last decade has been the shift from waterfall development methods to agile. Digital transformation is accelerating the adoption of agile because digital is the main portal to customers and the customers’ expectation is to get more – faster and easier.

In a world where a bad review can go viral in seconds, better apps and faster service have created a make or break scenario for vendors – customers are much less tolerant of bugs and issues. In addition, fast innovation by handset manufacturers and operating system makers has created a continuous stream of innovation opportunities e.g. easier, more secure authentication using biometric support and new workflow opportunities, such as using pictures as part of the insurance claims process. Agile makes it possible to increase velocity whilst ensuring quality, but it brings some challenges too. One hurdle when making this move has been getting to grips with the concept of iterative development.

For waterfall organisations, the accepted way of working has been to design and lay out the entire project prior to beginning development. Of course, this might entail months of requirement gathering and design sessions before a team could even begin working on code and during this lengthy process user requirements may change: what was once appropriate no longer fits, resulting in wasted efforts. Agile processes have caused a complete change in direction, helping teams move to focused, small and iterative work and, as a result, allowing them to develop and deploy services quicker than before.

Of course, the shift from waterfall to agile, which went against developers’ previous training and experience, caused apprehension for many. This is normal and was expected. But the clear benefits of the agile methodology helped developers to immediately recognise its value.

And, for us, the primary value of agile is speed. Agile creates fast feedback opportunities between developers and users, enabling faster course correction and improved quality.

Indeed, it’s this need for speed and fast feedback which is a driving force behind many of the changes we’ve seen in software development – and it’s the reason, too, for development teams’ responsibilities expanding exponentially over the last decade. Where testers, developers and quality assurance (QA) teams once had distinctly different jobs to do, the functions are quickly merging and dev teams are now tasked both with making apps and services and also ensuring they are of the highest quality.Q

The need for training

The developer now wears many ‘hats’ and, of course, the problem for some is that these hats aren’t necessarily the ones they want to wear, or even signed up to wear! As well as writing code, developers find themselves focused on enhancing and, improving customer experience, boosting service efficiencies and lowering costs. All vital jobs of course, but very different from what they were initially trained to do.

Let’s take microservices as an example. The deployment of microservices (the loosely coupled, autonomous, fine-grained services which many commentators see as the holy grail of modern development), is a sea change for many. Deploying many microservices to a shifting number of users means that app deployment is getting much more complicated, and so no longer falls under a pure-play developer’s remit. Instead, it becomes the role of a wider, frequently shared, DevOps function, which has responsibility for the whole product development and production lifecycle, and which is the function where many developers now sit.

So, in this world, where job functions are increasingly fluid and complex, it’s important that organisations are committed to upskilling their developers and sharpening skill sets towards the goal of automating key activities during the DevOps pipeline. As always, speed is crucial and teams must focus their efforts on improving the processes that are slowing them down.

We believe that leadership is a critical aspect of helping developers seamlessly transition to what essentially constitutes a new job role. Diverse teams and personas call for strong leadership as a unifying force, and a leader’s active role in affecting change is crucial. Of course, part of leadership is to enforce stringent metrics and KPIs which help to keep everyone on track. The continuous measurement of both development productivity (the number of user stories per sprint) and quality measurements (the number of defects per release) are imperative.

The need for testing

In the past, we saw developers follow a linear process when it came to testing. Committing code without testing, or with minimal testing, wasn’t a problem when QA teams were on hand to catch issues and fix them. But today the QA phase has gone (at least in mature shops), and the iterative development process that is the hallmark of the new DevOps world means that testing and quality assurance needs to happen in tandem with development.

And, just as the developer’s role is changing, so too have our ways of measuring: not only our progress but also the effectiveness and performance of our code, tools and processes. This increased visibility is a good thing and helps to catch issues early. And, when the developer ‘owns the quality’, continuous testing becomes absolutely crucial; it’s key to improving quality, extending coverage, speeding up development, reducing costs, and saving man hours, and the de facto requirement to achieve the development speed objectives of modern development teams.

And it’s with this plethora of reasons that we now see so much emphasis put on test automation. The development process has been fundamentally changed – commit code is not complete without commit tests alongside it, and new test tools and platforms have been developed in order to help developers deliver quality code and stable automation tests.

Evolution for responsibilities

So, the development world has fundamentally changed over the last decade. And there’s little doubt any more that DevOps has evolved from being the bright, shiny object that everyone talks about, to be the necessary reality. Developers, as part of the DevOps function, now have significant responsibility in an enterprise. They own the availability and sustainability of the customer experience – and that can make or break a business.

But, we recognise that, for a developer, this merging of roles we’ve seen over the last decade can be tricky. As a developer, it may be that you have always been spending a fair amount of your time on the Ops side, but now it’s your responsibility to do both. And much more besides.

Of course, it’s difficult to have at least half your job consumed by doing something you didn’t necessarily go to school to learn, but this increased responsibility can be vital to the success of a business. In recognition that their role is a crucial one, it’s our responsibility as an industry to help developers cope with the change.

Written by Uzi Eilon, CTO, Perfecto (America)

The post The changing role of the developer: how to adapt in a fast moving world appeared first on DevOps Online North America.

The cloud-based platform employed Carus because of his twenty years experience within the technology industry, with a proven track record of profitably scaling companies.

CEO at Perfecto, Eran Yanic, said: “Michael has a proven track record of working with a wide variety of companies, from startups to enterprise-level organisations. With Michael spearheading our financial initiatives, we’re well equipped to extend our growth trajectory, continue to attract new talent, and position the company for future capital events.

“We’re thrilled to welcome Michael to the team. His experience makes him the perfect candidate to own our long-term financial outlook and guide our profitable growth.”

Carus’s main role will be to support Perfecto teams as they optimise DevOps pipelines by enabling continuous delivery.

Written from press release by Leah Alger

Read more from DevOps Online | Follow us on Twitter

The post Perfecto’s new Chief Financial Officer appeared first on DevOps Online North America.

Perfecto, a cloud based platform for web, mobile and IoT software testing revealed in a recent report ‘The Path to Releasing Confidently in DevOps,’ key practices that boost mobile app teams and enterprise web.

In a bid to attain efficient high velocity software development, the report found the average developer spends less than 5-10% of their time writing code; high performance teams release software to production 2-3 times per week; while low performers tend to release in large batches closer to once per quarter and high performance teams are 2 times more likely to receive early user interface (UI) feedback than mid-performance and low-performance teams.

Perfecto also announced that developers who deliver value to production users also tend to embed quality into all software stages at the delivery lifecycle, as high velocity software delivery is delivering frequently requires confidence in app quality.
“We’ve found that the average developer spends less than 50% of their time writing code,” said Carlo Cadet, Director of Product Marketing at Perfecto.

“What is perhaps most surprising is that high performance dev teams do not spend any more or less time coding than their peers. Instead, high performance teams spend 5 to 10% less time on overhead tasks like admin and waste and 5% more time writing tests. This greater focus on quality helps deliver value more successfully. Embedding quality into all stages of the software delivery lifecycle is a difference maker.”

The report consists of 478 respondents from development professionals, which was created from 6 February 2017 to 20 March 2017.

Written from press release by Leah Alger

The post Perfecto reveals trends in recent DevOps report appeared first on DevOps Online North America.

The European Software Testing Awards is an annual event held in Central London celebrating the very best in the software testing and QA industry. A total of 18 awards were handed out for diverse categories such as Best Agile Testing Project, Best Mobile Testing Project and Leading Vendor. Two special awards were also handed out on the night – Micro Focus European Software Testing Award and an Outstanding Achievement Award.

A full list of the winners can be found here.

Raising money and awareness at The European Software Testing Awards 2016

During the evening, Senior Fundraising Executive, Catherine Sykes from GOSH took the stage for a moment to spread awareness of the charity and their work.

Members of the 31 Media team then collected donations and sold prize draw tickets. After the Awards Ceremony concluded, Ms Sykes was able to hand over a brand new Amazon Echo to the lucky winner.

All proceeds from the prize draw were donated to the charity.

Supporting Great Ormond Street Hospital (GOSH)

31 Media, organisers of The European Software Testing Awards and the publishers of TEST Magazine are really proud to have worked with some great charities to make a positive difference over the years.

“Using our events, publications, and platforms to donate to incredible causes such as Great Ormond Street Hospitals Children’s Charity is crucial to us as a company. We constantly want to be building awareness around extremely important causes such as this one, and we’re just so thankful to everyone who donated at this year’s awards,” says Cecilia Rehn, General Manager and Editor at TEST Magazine. “We’re incredibly proud of the amount we all, collectively, managed to raise on the night!”

About GOSH

GOSH is one of the top five paediatric research hospitals in the world, treating children from all over the UK and abroad who are diagnosed with the most complex, life-threatening conditions. GOSH has over 50 specialities under one roof and receives over 240,000 patient visits every year from children all over the UK and abroad, who are suffering from the most complex, life-threatening conditions.

The charity exists to assist the hospital in delivering world-class care for its young patients and their families and to pioneer new treatments and cures for childhood illness. Support for the charity will help the hospital to save more lives, develop new treatments, build new facilities and provide outstanding care to children and their families.

About The European Software Testing Awards

The Headline Sponsor for The European Software Testing Awards 2016 is MICRO FOCUS, and category sponsors are AMDOCS and WORKSOFT. The sponsors for The European Software Testing Summit 2016 include AMDOCS, CA TECHNOLOGIES and PERFECTO, supported by TEST Magazine, Software Testing News and DevOps Online.

31 Media, organisers of The European Software Testing Awards and publishers of TEST Magazine would like to thank its sponsors for their support.

The post Over £3000 raised for GOSH at The European Software Testing Awards 2016 appeared first on DevOps Online North America.

It should be no surprise that the explosive growth of mobile devices has changed how people interact and engage with brands, each other and the outside world at large. Whether swiping right for your love match, using ApplePay at the checkout, shopping for your next weekend outfit, watching the latest Game of Thrones or simply browsing the web, the mobile app has become integral to the consumer experience.

Like all channels that came before it, succeeding on mobile is dependent on businesses creating a winning user experience. Business goals, customer satisfaction, positive app store reviews, word-of-mouth uptick and, of course, revenue are increasingly being driven through mobile. But in complex digital environments, testing just basic mobile app features under ideal conditions will fall short of delivering a great user experience.

When trying to deliver high quality mobile on a variety of operating systems and networks, it’s important to remember to not only test your apps functionalities, but also the right user environments.

Testing beyond functionality

Most businesses’ mobile app testing and quality assurance (QA) teams carefully test the “functional” aspects of their mobile apps. This means making sure that critical functions such as downloading and installing, logging in, search, payment processing, checkout, scrolling, etc. work. Whilst this is important, developers know that they can no longer create applications assuming they will be run from desktops with steady power supplies and stable broadband connections. By their nature, mobile apps will be used from different locations and under different conditions, forcing your teams to consider various factors and user conditions, such as:

• How frequently do your users use your app?
• Are your users always on Wi-Fi or cellular, or are they switching between the two?
• What other apps are interrupting their usage?
• What mobile devices are they using to access your app?
• How does your app perform under weak network conditions?

These unique usage patterns are critical to improving mobile quality and preventing user churn. And in today’s digital age, it’s essential to test for the real user conditions that your customers experience every day.

Testing in the user’s environment

To effectively test real user conditions, you need to be able to replicate the real environments, devices and conditions under which users will be using your app. In making user conditions part of your mobile app testing, there are three practical steps businesses will need to take:

1. Define the right users for your products

To be able to test under real user conditions, you must first identify what conditions your typical users will likely be using your app under. The first step is therefore to identify your target users – otherwise known as “personas”. While this sounds easy, accurate personas can only be created through market research and data analytics. As such, defining personas relies on consistent communication between engineering and business teams.

The next step is integrating these users and their traits into the test lab. This is usually the toughest task for engineering teams because mimicking real environment conditions is hard and requires ongoing lab maintenance.

Some of the core requirements needed to sustain a lab that tests for user conditions include:

• Ability to simulate various real life factors into the testing lab, such as network conditions, apps running in the background, phone call interruptions, etc.
• Support for different geographies to cover testing of relevant users in relevant countries
• Support for any device and any operating system to sustain lab coverage as market matures and continues to develop

2. Add user conditions to existing functional tests

The second step, once the lab is ready, is for DevTesters to add the relevant user conditions to existing functional tests in different ways, such as manual testing, data-driven testing or through automated test code. In most cases, the most efficient path will be through automation.

Businesses can implement user condition testing on top of functional test automation code by adding test code lines that define the target. This test code will automatically inherit user traits such as their name, the platforms and devices they use, their background apps, their location, their network conditions, etc.

Integrating these user traits into an existing continuous integration (CI) workflow and running many scenarios simultaneously will pay off in the form of shorter release cycles, increased test coverage and, ultimately, a better user experience.

3. Use reports to find and fix issues

Mobile app development and testing is a continuous process. Once your team has defined the target personas and implemented the test code correctly, they’ll need a detailed and actionable rich report to continuously drive improvements. Such a report should include network logs, device and app memory and battery consumption details, and timers showing how long it took certain actions to take place.

To have actionable data, these reports must include results for every test scenario and should enable teams to take action to fix any reported issues. To be able to consistently address bug fixes, the report should also compare the specific user condition tests to a “happy path” to show the contrast between real world conditions and ideal conditions.

By understanding your users’ conditions and taking practical steps with the right tools to test in real user environments will allow brands to ensure a high quality experience for their customers – regardless of where or under which circumstances they are engaging with your app and brand.

Edited for web by Jordan Platt

The post How to test your mobile apps in the right environments appeared first on DevOps Online North America.

Taking the stage on first day will be Stephen Williams, VP Engineering, Ticketmaster International, who will be presenting on a DevOps case study.

“My talk is on how we defined a DevOps strategy and programme within Ticketmaster, which allows flexible planning of how geographically distributed teams progress,” Williams explained.

“The second half of the talk will present some of the lessons we’ve learnt and the challenges to be expected when implementing a DevOps transformation programme.”

Stephen Williams, VP Engineering, Ticketmaster International, will be presenting on: ‘DevOps – Strategy, Rollout & Lessons Learnt’ at The National DevOps Conference.

DevOps is critical to business wishing to reduce time to market

“DevOps (bringing Dev and Ops together) is really an extension to agile (bringing Dev and QA together), being responsive to business needs and enabling change in direction (pivoting) quickly where required and therefore being more innovative,” Williams said.

DevOps enables this by allowing changes to be deployed into production fast whilst reducing waste and risks. By going faster the organisation is able to get faster feedback on the impacts of the changes.

“You can liken this to the difference between having 1 shot on goal versus having 100 shots on goal in the same time frame. You’re more likely to score a goal when you have 100 shots. Therefore as we go faster the more innovative we can be,” Williams added.

What can organisations learn from Ticketmaster’s DevOps journey?

Williams explained what delegates at The National DevOps Conference will be able to take away from his presentation.

“My key takeaways are: Standardised where it makes sense, so teams can communicate, and share learnings and skills. Focus on small wins and removing waste. Measure what you can and show the impact of the impact of the changes, then expand to more meaningful measurements. Organise teams to include cross-functional skills Dev-QA-Sec-Ops and empower them to automate wherever possible,” Williams said.

“Finally be prepared for a transformational journey. There’s no short cut, and although the technical change is hard, be prepared that organisational change could be harder.”

Why attend The National DevOps Conference?

The National DevOps Conference provides an ideal environment for learning, networking and developing skills.

Williams shares his advice on attending the conference: “It’s always great at a conference such as this because you have such a wide array of people from different businesses, different viewpoints and different business domains and problems. Understanding other people’s approaches and the problems they’ve solved can only be a good thing to expand and strengthening one’s own knowledge and understanding of the DevOps domain space.

“It’s also a great opportunity to network, making those relationships that can help you draw on expertise post conference, as well as other potential business opportunities.”

Software testing and QA streams

In addition to bringing excellence, best practices and practical advice from the DevOps scene, The National DevOps Conference will include two software testing & QA streams for those interested in learning about testing’s role in the software development lifecycle.

These streams will include presentations from the public, retail, financial and gaming sectors.

The National DevOps Conference is sponsored by Hewlett Packard Enterprise, Perfecto, iSQI, Neotys, Delphix, CAPITA, nFocus, Valuelabs, Applause, Infostretch, IBM, Parasoft, BCS, Polarion, iTrinegy, CA Technologies, SMT Software Services, and Undo Software.

31 Media, organisers of The National DevOps Conference and publishers of TEST Magazine would like to thank its sponsors for their support.

Written by Cecilia Rehn.

The post Challenges to be expected when implementing a DevOps transformation program – Ticketmaster International’s experience appeared first on DevOps Online North America.