{"id":13519,"date":"2018-07-20T15:40:35","date_gmt":"2018-07-20T14:40:35","guid":{"rendered":"http:\/\/www.devopsonline.co.uk\/?p=13519"},"modified":"2018-07-20T15:40:35","modified_gmt":"2018-07-20T14:40:35","slug":"the-relationship-between-continuous-testing-non-functional-testing","status":"publish","type":"post","link":"https:\/\/devopsnews.online\/the-relationship-between-continuous-testing-non-functional-testing\/","title":{"rendered":"The relationship between continuous testing &#038; non-functional testing"},"content":{"rendered":"<p>Firstly, let\u2019s understand what is meant by continuous testing. The following definition from Wikipedia, is good enough for our purposes: \u2018continuous testing is the process of executing automated tests as part of the software delivery pipeline in order to obtain feedback on the business risks associated with a software release candidate as rapidly as possible\u2019.<\/p>\n<p>There are some key points that come out of this definition:<\/p>\n<ul>\n<li>Automated tests: firstly, whatever type of testing you may be looking to execute, you generally can\u2019t deliver continuous testing in any meaningful way without using automation<\/li>\n<li>Feedback: secondly, the point of continuous testing is to provide regular feedback to the business on the impact of change as a result of Continuous Integration and continuous delivery<\/li>\n<li>Business risk: finally, increased velocity in release deployment increases the risk of introducing functional error, performance and capacity regression to a new deployment. Introducing CT at each stage of the software development life-cycle (SDLC) helps mitigate the risk of this occurring.<\/li>\n<\/ul>\n<p>It would be true to say that continuous testing, continuous integration and continuous deployment are all part of DevOps, enabling the feedback loop that drives an informed, agile and robust approach to enterprise software development and deployment.<\/p>\n<h2>What is non-functional testing?<\/h2>\n<p>Non-functional testing refers to testing the way a system operates rather than specific system behaviour. Whilst non-functional testing can also encompass security and acceptance testing, the most common use case is testing software performance: reliability and scalability under load.<\/p>\n<p>Let\u2019s look in more detail at the key points from our definition of continuous testing and how they align with non-functional testing requirements.<\/p>\n<h2>Automated testing<\/h2>\n<p>Delivering non-functional testing requires automation and there is a wide range of tooling available, both licensed and open source. Historically this has applied mainly to system testing carried out late in the SDLC, leaving little time to correct any problems uncovered by testing.<\/p>\n<p>Non-functional test automation now increasingly extends into Dev, where the move to modular or microservice-based architecture simplifies testing at the component level. This is often accomplished using open source tooling, such as JMeter (jmeter.apache.org) which has a powerful scripting engine and is easy to deploy.<\/p>\n<p>Continuous non-functional testing is relatively simple to integrate into the Dev process, either ad-hoc or as part of the overnight build\/test\/deploy. This allows trending of component performance across builds and early detection and correction of coding changes that inadvertently cause performance regression.<\/p>\n<p>The increasingly common use of highly scalable cloud-based test environments leveraging endpoint mock, stub or service virtualisation can allow developers to test component performance in sprint against production SLA\u2019s, something unheard of even five years ago.<\/p>\n<h2>Feedback<\/h2>\n<p>Non-functional testing that is limited to system testing late in the SDLC is poorly placed to provide timely feedback to the Business on application performance. The feedback loop that is a key goal of implementing DevOps needs to function right across the SDLC to be truly effective.<\/p>\n<p>Ideally continuous non-functional testing should be implemented in the following areas:<\/p>\n<ul>\n<li>development<\/li>\n<li>component and integrated component testing<\/li>\n<li>QA<\/li>\n<li>system testing<\/li>\n<li>production<\/li>\n<li>scheduled BAU testing.<\/li>\n<\/ul>\n<p>Feedback data can take several forms but the most important consideration is visibility of application performance, good and bad, to relevant stakeholders. This means rapid feedback to Dev of performance-related bugs so they can be corrected and, if necessary, a new release prepared \u2013 including the dashboarding of test results demonstrating that an application is meeting or failing to meet the assigned performance goals.<\/p>\n<p>Many organisations leverage full-stack monitoring solutions like ELK and Graphite \/ Grafana to visualise application behaviour. It is vitally important that performance data is included to inform stakeholder groups tasked with key responsibilities, such as approving Dev team work items and application release candidates.<\/p>\n<h2>Business risk<\/h2>\n<p>Mitigating business risk is all about ensuring that core applications remain available, functionally stable and performant to end-users regardless of load. Implementing continuous non-functional testing in the areas previously described helps to make this happen by maintaining and improving the quality of release.<\/p>\n<p>The business impact of problems uncovered at each stage of testing can be quickly assessed, prioritised and corrected. This promotes early discovery of problems so the number and severity of non-functional software bugs naturally decrease and shifts left, benefiting the speed and quality of delivery.<\/p>\n<p>It is equally important to understand the ongoing performance of core software applications post-deployment. Regularly scheduled non-functional testing of production systems can help detect creeping performance regressions, often the cumulative result of many small releases.<\/p>\n<p>Scheduled BAU testing also provides confidence that production deployments are performance-ready in advance of peak events like Black Friday and Cyber Monday.<\/p>\n<p>Written by Independant Consultant,\u00a0Ian Molyneaux<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Ian Molyneaux, Independent Consultant, talks in-depth about how continuous testing has a close relationship with non-functional testing, together driving an informed, agile and robust approach to enterprise software development and deployment<\/p>\n","protected":false},"author":2,"featured_media":13520,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"content-type":"","pmpro_default_level":"","footnotes":""},"categories":[2],"tags":[2984,558,2985,679,2982,1368,395,1938,2412,2983],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v22.5 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>The relationship between continuous testing &amp; non-functional testing<\/title>\n<meta name=\"description\" content=\"Ian Molyneaux, Independent Consultant, talks in-depth about how continuous testing has a close relationship with non-functional testing\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/devopsnews.online\/the-relationship-between-continuous-testing-non-functional-testing\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"The relationship between continuous testing &amp; non-functional testing\" \/>\n<meta property=\"og:description\" content=\"Ian Molyneaux, Independent Consultant, talks in-depth about how continuous testing has a close relationship with non-functional testing\" \/>\n<meta property=\"og:url\" content=\"https:\/\/devopsnews.online\/the-relationship-between-continuous-testing-non-functional-testing\/\" \/>\n<meta property=\"og:site_name\" content=\"DevOps Online North America\" \/>\n<meta property=\"article:published_time\" content=\"2018-07-20T14:40:35+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/devopsnews.online\/wp-content\/uploads\/2018\/07\/iStock-913223940.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"788\" \/>\n\t<meta property=\"og:image:height\" content=\"443\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"DevOps Online\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@DevOpsAmerica\" \/>\n<meta name=\"twitter:site\" content=\"@DevOpsAmerica\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"DevOps Online\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/devopsnews.online\/the-relationship-between-continuous-testing-non-functional-testing\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/devopsnews.online\/the-relationship-between-continuous-testing-non-functional-testing\/\"},\"author\":{\"name\":\"DevOps Online\",\"@id\":\"https:\/\/devopsnews.online\/#\/schema\/person\/de52473fff111f14d90763193184cb1e\"},\"headline\":\"The relationship between continuous testing &#038; non-functional testing\",\"datePublished\":\"2018-07-20T14:40:35+00:00\",\"dateModified\":\"2018-07-20T14:40:35+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/devopsnews.online\/the-relationship-between-continuous-testing-non-functional-testing\/\"},\"wordCount\":779,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/devopsnews.online\/#organization\"},\"image\":{\"@id\":\"https:\/\/devopsnews.online\/the-relationship-between-continuous-testing-non-functional-testing\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/devopsnews.online\/wp-content\/uploads\/2018\/07\/iStock-913223940.jpg\",\"keywords\":[\"BAU testing\",\"continuous testing\",\"CT\",\"development\",\"non-functional\",\"PRODUCTION\",\"QA\",\"SDLC\",\"sla\",\"system testing\"],\"articleSection\":[\"Featured\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/devopsnews.online\/the-relationship-between-continuous-testing-non-functional-testing\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/devopsnews.online\/the-relationship-between-continuous-testing-non-functional-testing\/\",\"url\":\"https:\/\/devopsnews.online\/the-relationship-between-continuous-testing-non-functional-testing\/\",\"name\":\"The relationship between continuous testing & non-functional testing\",\"isPartOf\":{\"@id\":\"https:\/\/devopsnews.online\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/devopsnews.online\/the-relationship-between-continuous-testing-non-functional-testing\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/devopsnews.online\/the-relationship-between-continuous-testing-non-functional-testing\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/devopsnews.online\/wp-content\/uploads\/2018\/07\/iStock-913223940.jpg\",\"datePublished\":\"2018-07-20T14:40:35+00:00\",\"dateModified\":\"2018-07-20T14:40:35+00:00\",\"description\":\"Ian Molyneaux, Independent Consultant, talks in-depth about how continuous testing has a close relationship with non-functional testing\",\"breadcrumb\":{\"@id\":\"https:\/\/devopsnews.online\/the-relationship-between-continuous-testing-non-functional-testing\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/devopsnews.online\/the-relationship-between-continuous-testing-non-functional-testing\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/devopsnews.online\/the-relationship-between-continuous-testing-non-functional-testing\/#primaryimage\",\"url\":\"https:\/\/devopsnews.online\/wp-content\/uploads\/2018\/07\/iStock-913223940.jpg\",\"contentUrl\":\"https:\/\/devopsnews.online\/wp-content\/uploads\/2018\/07\/iStock-913223940.jpg\",\"width\":788,\"height\":443,\"caption\":\"continuous testing\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/devopsnews.online\/the-relationship-between-continuous-testing-non-functional-testing\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/devopsnews.online\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"The relationship between continuous testing &#038; non-functional testing\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/devopsnews.online\/#website\",\"url\":\"https:\/\/devopsnews.online\/\",\"name\":\"DevOps Online North America\",\"description\":\"by 31 Media Ltd.\",\"publisher\":{\"@id\":\"https:\/\/devopsnews.online\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/devopsnews.online\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/devopsnews.online\/#organization\",\"name\":\"DevOps Online North America\",\"url\":\"https:\/\/devopsnews.online\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/devopsnews.online\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/devopsnews.online\/wp-content\/uploads\/2020\/03\/DevOpsOnline_email.png\",\"contentUrl\":\"https:\/\/devopsnews.online\/wp-content\/uploads\/2020\/03\/DevOpsOnline_email.png\",\"width\":198,\"height\":64,\"caption\":\"DevOps Online North America\"},\"image\":{\"@id\":\"https:\/\/devopsnews.online\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/x.com\/DevOpsAmerica\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/devopsnews.online\/#\/schema\/person\/de52473fff111f14d90763193184cb1e\",\"name\":\"DevOps Online\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/devopsnews.online\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/cf0ba37fb1f8baf226b40986afbe7f9f?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/cf0ba37fb1f8baf226b40986afbe7f9f?s=96&d=mm&r=g\",\"caption\":\"DevOps Online\"},\"url\":\"https:\/\/devopsnews.online\/author\/test-magazine\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"The relationship between continuous testing & non-functional testing","description":"Ian Molyneaux, Independent Consultant, talks in-depth about how continuous testing has a close relationship with non-functional testing","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/devopsnews.online\/the-relationship-between-continuous-testing-non-functional-testing\/","og_locale":"en_US","og_type":"article","og_title":"The relationship between continuous testing & non-functional testing","og_description":"Ian Molyneaux, Independent Consultant, talks in-depth about how continuous testing has a close relationship with non-functional testing","og_url":"https:\/\/devopsnews.online\/the-relationship-between-continuous-testing-non-functional-testing\/","og_site_name":"DevOps Online North America","article_published_time":"2018-07-20T14:40:35+00:00","og_image":[{"width":788,"height":443,"url":"https:\/\/devopsnews.online\/wp-content\/uploads\/2018\/07\/iStock-913223940.jpg","type":"image\/jpeg"}],"author":"DevOps Online","twitter_card":"summary_large_image","twitter_creator":"@DevOpsAmerica","twitter_site":"@DevOpsAmerica","twitter_misc":{"Written by":"DevOps Online","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/devopsnews.online\/the-relationship-between-continuous-testing-non-functional-testing\/#article","isPartOf":{"@id":"https:\/\/devopsnews.online\/the-relationship-between-continuous-testing-non-functional-testing\/"},"author":{"name":"DevOps Online","@id":"https:\/\/devopsnews.online\/#\/schema\/person\/de52473fff111f14d90763193184cb1e"},"headline":"The relationship between continuous testing &#038; non-functional testing","datePublished":"2018-07-20T14:40:35+00:00","dateModified":"2018-07-20T14:40:35+00:00","mainEntityOfPage":{"@id":"https:\/\/devopsnews.online\/the-relationship-between-continuous-testing-non-functional-testing\/"},"wordCount":779,"commentCount":0,"publisher":{"@id":"https:\/\/devopsnews.online\/#organization"},"image":{"@id":"https:\/\/devopsnews.online\/the-relationship-between-continuous-testing-non-functional-testing\/#primaryimage"},"thumbnailUrl":"https:\/\/devopsnews.online\/wp-content\/uploads\/2018\/07\/iStock-913223940.jpg","keywords":["BAU testing","continuous testing","CT","development","non-functional","PRODUCTION","QA","SDLC","sla","system testing"],"articleSection":["Featured"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/devopsnews.online\/the-relationship-between-continuous-testing-non-functional-testing\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/devopsnews.online\/the-relationship-between-continuous-testing-non-functional-testing\/","url":"https:\/\/devopsnews.online\/the-relationship-between-continuous-testing-non-functional-testing\/","name":"The relationship between continuous testing & non-functional testing","isPartOf":{"@id":"https:\/\/devopsnews.online\/#website"},"primaryImageOfPage":{"@id":"https:\/\/devopsnews.online\/the-relationship-between-continuous-testing-non-functional-testing\/#primaryimage"},"image":{"@id":"https:\/\/devopsnews.online\/the-relationship-between-continuous-testing-non-functional-testing\/#primaryimage"},"thumbnailUrl":"https:\/\/devopsnews.online\/wp-content\/uploads\/2018\/07\/iStock-913223940.jpg","datePublished":"2018-07-20T14:40:35+00:00","dateModified":"2018-07-20T14:40:35+00:00","description":"Ian Molyneaux, Independent Consultant, talks in-depth about how continuous testing has a close relationship with non-functional testing","breadcrumb":{"@id":"https:\/\/devopsnews.online\/the-relationship-between-continuous-testing-non-functional-testing\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/devopsnews.online\/the-relationship-between-continuous-testing-non-functional-testing\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/devopsnews.online\/the-relationship-between-continuous-testing-non-functional-testing\/#primaryimage","url":"https:\/\/devopsnews.online\/wp-content\/uploads\/2018\/07\/iStock-913223940.jpg","contentUrl":"https:\/\/devopsnews.online\/wp-content\/uploads\/2018\/07\/iStock-913223940.jpg","width":788,"height":443,"caption":"continuous testing"},{"@type":"BreadcrumbList","@id":"https:\/\/devopsnews.online\/the-relationship-between-continuous-testing-non-functional-testing\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/devopsnews.online\/"},{"@type":"ListItem","position":2,"name":"The relationship between continuous testing &#038; non-functional testing"}]},{"@type":"WebSite","@id":"https:\/\/devopsnews.online\/#website","url":"https:\/\/devopsnews.online\/","name":"DevOps Online North America","description":"by 31 Media Ltd.","publisher":{"@id":"https:\/\/devopsnews.online\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/devopsnews.online\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/devopsnews.online\/#organization","name":"DevOps Online North America","url":"https:\/\/devopsnews.online\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/devopsnews.online\/#\/schema\/logo\/image\/","url":"https:\/\/devopsnews.online\/wp-content\/uploads\/2020\/03\/DevOpsOnline_email.png","contentUrl":"https:\/\/devopsnews.online\/wp-content\/uploads\/2020\/03\/DevOpsOnline_email.png","width":198,"height":64,"caption":"DevOps Online North America"},"image":{"@id":"https:\/\/devopsnews.online\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/x.com\/DevOpsAmerica"]},{"@type":"Person","@id":"https:\/\/devopsnews.online\/#\/schema\/person\/de52473fff111f14d90763193184cb1e","name":"DevOps Online","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/devopsnews.online\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/cf0ba37fb1f8baf226b40986afbe7f9f?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/cf0ba37fb1f8baf226b40986afbe7f9f?s=96&d=mm&r=g","caption":"DevOps Online"},"url":"https:\/\/devopsnews.online\/author\/test-magazine\/"}]}},"_links":{"self":[{"href":"https:\/\/devopsnews.online\/wp-json\/wp\/v2\/posts\/13519"}],"collection":[{"href":"https:\/\/devopsnews.online\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devopsnews.online\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devopsnews.online\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/devopsnews.online\/wp-json\/wp\/v2\/comments?post=13519"}],"version-history":[{"count":0,"href":"https:\/\/devopsnews.online\/wp-json\/wp\/v2\/posts\/13519\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devopsnews.online\/wp-json\/wp\/v2\/media\/13520"}],"wp:attachment":[{"href":"https:\/\/devopsnews.online\/wp-json\/wp\/v2\/media?parent=13519"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devopsnews.online\/wp-json\/wp\/v2\/categories?post=13519"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devopsnews.online\/wp-json\/wp\/v2\/tags?post=13519"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}