Successful software testing outsourcing goes hand in hand with QA vendor performance management. Here’s how to properly organise this process.

Successful software testing outsourcing is a challenging but manageable process. And QA vendor performance management is definitely a thing not to be neglected as choosing a top-notch QA and testing vendor is not enough.

Only 39% of leading IT companies are successful in managing their vendors.

You can’t get the QA services you want without the sufficient management efforts from your side, so both you and your vendor need to cooperate efficiently to reach the project goals. So, let’s clear out why QA vendor performance management is vital, who is to manage it, and how it should be carried out.

Why QA vendor performance management?

Let’s consider some common risks of neglected or ineffective management of software testing vendor performance that can influence your general QA outsourcing efficiency:

• A significant number of missed bugs resulting from a poor understanding of software requirements by the testing team
• Incoherent bug description may become a problem, if you don’t talk over a conventional bug reporting procedure or bug tracking tools with your vendor
• Excessive costs. This issue may arise if the testing strategy is unbalanced, implies unreasonable testing types or human resources allocation, lacks relevant test automation, or your vendor reports the activities that were performed only partially (or not performed at all) as completed
• A vendor doesn’t improve their performance. If they don’t get your feedback stating that there is space for the performance improvement (optimising testing teams, providing more coherent bug reports or introducing the relevant testing types), they may not even consider it
• Vendor lock-in may be a nasty side effect in case your vendor uses legacy software testing tools and neglects testing documentation
• Difficulties with contract management may arise if vendor management processes and vendor performance assessment criteria aren’t agreed in the contract, so it’s highly complicated for your vendor to live up to relevant KPIs without even knowing them.

Luckily, QA vendor performance management is a viable practice to reduce to a minimum these and more risks on the condition that it’s thoroughly organised.

Who is to perform it?

You need to employ or assign a QA manager to carry out the activities implied by software testing vendor management. The tricky point here is that your professional is to be experienced in QA and at the same time be acquainted with general vendor management principles.

These principles will be helpful for practices like risks and contract management, KPIs implementation. Still, QA vendor performance management is to meet actual testing outsourcing challenges (the adequacy of testing strategy and plans or constrained communication between development and testing teams).

So, the QA manager needs to regularly review and analyse testing documentation (a testing strategy, test plans, test cases, and test reports). Besides, QA know-hows, best practices and quality metrics should help them professionally check and evaluate software testing performance results and provide practical suggestions on how to improve the QA vendor’s performance.

Normally, one in-house QA manager is enough to manage a team of 20 to 50 outsourced testers. If you have a number of projects or your project implies the involvement of more than 50 testers, you may need to assign more QA managers for effective vendor management

How to lay the foundation for your QA vendor performance management?

The terms of QA vendor management should be agreed in the contract with your software testing vendor to know for sure what KPIs and service provision requirements they are to live up to. You should also take a note of financial terms, as inattention to them can seriously imped contract management by not allowing you to pay in accordance with the vendor’s performance results. It would be most secure for you to agree on a performance-based payment part provided only if the contract requirements are met. In addition, if your vendor’s performance results are poor (too many missing defects, the vendor’s bug description is so incoherent that developers have to guess what was meant, etc.) and they are reluctant to improve, the contract is to allow clear and easy termination options.

For a large-scale QA outsourcing project or a number of them, it makes sense to divide the whole scope of QA and testing services into manageable parts and formalize them in several service level agreements (SLAs). There can be a separate SLA for each project or project part. An SLA may include the subset of services to be rendered, the time limits for their execution, the parties’ responsibilities, payment conditions based on the vendor’s meeting SLA goals, acceptable KPI results and termination options.

How to perform QA vendor management?

To manage the vendor efficiently, the QA manager should continuously carry out the following activities:

• Monitor the testing team’s performance and output using the conventional bug tracking and project management tools.
• Review test documentation at least once a month.
• Ensure the SLA terms and requirements are followed and met.
• Organize updates of software requirements.
• Provide judgment in disputable situations able to hinder the testing performance.
• Mediate the problem solving process.
• Provide the testing team with continuous feedback on their performance.
• Suggest the possibilities to improve vendor performance.

The feedback provided to your QA vendor should be based on performance assessment results. It’s essential to evaluate the vendor performance regularly to be sure software testing outsourcing meets your expectations.

The results of vendor performance assessment will help to find the ways to enhance their performance, decide on a vendor replacement or a multivendor strategy in case of need.

Your QA manager should create the actual set of KPIs embracing specific project requirements and apt challenges.

They may find helpful the following common performance criteria for their future KPIs:

• The number of missed bugs
• The quality of the defects’ description
• The correspondence between the reported vendor’s efforts and actual outcomes
• The quality of the testing documents
• The relevancy of implemented frameworks and tools
• The sufficiency of the vendor’s human resources.

Conclusion

QA vendor performance management is an inseparable part of beneficial software testing outsourcing, as your vendor needs to receive constant feedback in order to meet your expectations and improve their performance.

Relevant contract conditions and SLAs can provide solid legal ground for regular vendor performance assessment and comprehensive management.

And a full-time QA manager responsible for vendor management will help to carry out this process professionally.

Andrei Mikhailau, SW testing director, ScienceSoft