Assessing Productivity of QA Teams

This post was originally published on this site

As a team lead, your job is to maximize your Quality Assurance (QA) team productivity.  

The question is: how do you measure QA team performance? 

You need to objectively measure QA metrics first, and then share progress with the management. If you don’t measure productivity of your QA team, then chances are that you wouldn’t know if things have started moving south. You want to know who is actually working in your team and bringing about the desired results, as well as be cognizant of those who are not pulling their weight, and consequently, not producing expected results.

Having this knowledge allows you to give appropriate appreciation and feedback, so that your team members can work accordingly and improve the areas they are lacking in. It’s just like looking at gauges and back mirrors while driving a car and steering it according to the situation—in other words, closely monitoring productivity keeps you abreast of which areas need to be worked upon and how to resolve them accordingly.  

Furthermore, you need to establish the QA metrics for good performance with the management, so that you can review it periodically. Having an established criterion is helpful as it allows you to gauge performance of all QA team members according to the same metrics. That being said, you can still set different goals and agendas for team members, based on their individual performance levels.  

Lets see what Kualitatem’s Service Delivery Manager, Ishfaq Zia, a QA enthusiast, had to say about the way he manages and oversees his QA team using performance metrics:  

“It’s important to get your clients on board with your process from the start. Once the expectations are understood, our approach is to prepare a high-level demo and then highlight how we solve their problem. We usually are on a time crunch and there’s a lot of work involved, especially if it is something the team isn’t familiar with. Having metrics in place, the team completes the breakdown of their long term goals into weekly goals which then translates into meeting timelines.” 

What is productivity? 

Simply put, productivity of any QA team is, no surprises here, portrayed by the highest quality of software within timelines. After all, “Quality Assurance” is called as such for a reason—it refers to methods that indicate the topmost quality of a product or service, with minimum number of defects or issues.  

Now comes the question: how do we gauge the software quality? 

This translates into the following main areas: 

  1. Thoroughness of test cases: The test cases should be thoroughly covered, as this is linked with high software quality.  
  1. Speed
  • Test execution speed 
  • Defect fixing speed 

   Greater speed, for both test execution and defect fixing, depicts higher productivity. 

  1. Product Quality: There should be no defects found in the production stage. 

This is what Ishfaq had to say regarding reducing the number of bugs before production:  

“Exhaustive or complete testing is impossible. Occasionally, a client comes around after several rounds of our own testing and locates a major bug that we should’ve picked on. While there is always a possibility, ensuring a complete understanding of the product and utilizing proper tools is essential to negate this risk as much as possible”.   

Furthermore, a test management tool, such as Kualitee, should tell how you are doing as team and as an individual, both. After all, a significant part of individual effort contributes to team effort, so both go hand in hand to some extent.  

When do you know when productivity is good? 

Research industry benchmarks for your industry/project 

Do thorough research on your industry standards and benchmarks, and set goals according to them. Remember that your aim is to rise in a competitive industry, which is why it becomes important to remain cognizant of these benchmarks.  

Take input from stakeholders 

Your stakeholders include a wide pool of people–including team members, managers, clients, etc. Instead of getting feedback from only one party, be open to receiving input from all the stakeholders. This way, you can take note of points that you may not have realized on your own as they are not relevant to you directly, or that may have slipped your eye. Thus, be open and receptive to hearing suggestions and feedback.   

Set expectations early regarding good and bad performance 

Instead of only setting expectations regarding your QA team members’ productivity, for yourself, you can also convey those expectations to them. Having that knowledge will motivate them to work more efficiently or however required, as they would strive to live up to those expectations, as well as avoid those factors or behaviours that indicate deteriorating performance.  

Using KPIs to measure productivity

Productivity of your QA team can be assessed by measuring Key Performance Indicators (KPIs). 

Defining KPIs helps monitor the quality of software in terms of the three factors discussed earlier; I.e. test coverage, speed, and number of defects found. Setting the appropriate KPIs for your team will help you gauge progress, while also being able to gauge team productivity more easily. 

Testing in Production: Pros and Cons

This post was originally published on this site

Testing in production refers to the act of frequently performing tests in a live environment, after production deployment. It is not a substitute for any other type of testing, and the significance of the QA team still remains intact.  

Production testing has both its advantages and disadvantages, which may explain why some testers approach it with hesitancy, or have done so in the past. Let’s have a look at these pros and cons, below.  


Real World Accuracy 

Testing production systems are more geared towards reality. On the other hand, testing in non-production environments like development, quality assurance, and staging systems, does have value but it does not mirror the real world. This is because they are running on different codes, server operating systems, and patch levels. Often, testing in the two environments will yield starkly different results, which are particularly noticeable in penetration testing, as configurations vary at the application and server levels. 

Verifying Functionality 

Smoke and sanity tests are mostly performed in production environments. Smoke testing checks if the product’s main functionalities are working correctly. Sanity testing verifies if bugs have been fixed and the functionality of the product is as expected. This enhances the likelihood of improved service.  

User Experience 

Performing usability testing in a production environment can give businesses an understanding of the user experience. A/B testers offer two versions of a product to users and find statistics to determine which version performs better. This A/B testing method is ideal to test for new features, whose market value is unknown. 

Disaster Prevention 

No matter how much testing is done in development, it is inevitable that bugs will reach the production stage. If the DevOps team is prepared to handle defects at the production stage, the recovery time will be faster.

Hence, testing in production helps in averting disastrous situations—by taking testing a step ahead of basic prevention techniques, to monitoring methods that detect and fix bugs continuously. The application gains resilience and can recover from unexpected chaos without loss of functionality or data. 

Complete test coverage 

As conditions of a production environment cannot be fully replicated, complete test coverage is not possible before product deployment. Due to data and traffic limitations, many edge case tests in staging environments will be missed out. Testing in production will give full test coverage via continuous monitoring. With production data, every possible scenario can be detected. Also, developers often make changes in applications and deploy to production, skipping the Quality Assurance verification. These changes create bugs in the application. Testing in production with software testing tools removes the risks caused by these changes. 


Irresponsibility in testing 

Testing in production improves the overall testing process but it is not meant to lessen the tester’s responsibility for testing in other environments. As production testers share the responsibility of testing, QA testers may develop an unmotivated approach and not utilize the saved time towards other types of testing. Testing in production will only be effective if team members collaborate fully and not neglect other types of testing. 

Test timings 

If a load test is run on the production environment during business hours, it increases the chances for poor user experiences. Experts recommend running tests in production environments during times that do not impact a lot of users; for example, during maintenance hours. Testers get a smaller time slot to perform the tests. As user experience is tied with profit, a tester not adhering to the recommended timings can lower the business revenue. 

Chaos Monkey 

Chaos Monkey is a code with which chaos is created on purpose in a production environment. It is a technique to verify if production is working as expected. However, a code that injects failures deliberately can cause problems in an environment like damaging a web server. Chaos monkey should be used carefully when developing a resilient application to ensure application tiers are not killed. 


If conducted properly in a way that enhances efficiency, testing in production is a fruitful opportunity. Its benefits bring the product team closer to the end-users. However, the scope of testing in production has not been fully tapped into yet. Thus, exploring the potential of testing in production will become clearer gradually, as it is carried out more. 

Top 5 Responsibilities of QA Managers in an Agile Environment

This post was originally published on this site

Quality Assurance (QA) is different from software development. A Quality Assurance manager needs to have a certain skillset, while being abreast of the latest methodologies, technologies, and trends, in order to be successful in their profession. 

Keeping the above in mind, we present to you a list of the top 5 responsibilities of a QA manager, especially tailored to an agile environment

1. Set standards to be followed 

A good QA manager is not controlling; he or she gives their agile team members some responsibility and independence to manage the work by themselves. However, this does not mean that there should be no set standards in place, for the entire team to follow. 

The organization as a whole is expected to work as a coherent unit. It should have a few pre-defined standards like: 

  • Which Software Testing tools to use, and how to use them 
  • Which Testing Methodologies to apply 
  • How to deploy effective defect management, e.g. defining defect statuses 
  • Strategies to recognize the state of a build, and working accordingly. 

A QA manager works with the project managers to decide upon the standards that are to be utilized. This ensures that all team members are following similar techniques and being trained according to the same sets of rules. The managers then monitor progress and quality during the development process. These guidelines are used by the R&D team to incorporate QA processes within their development teams. 

2. Build teams wisely 

All development teams work according to their own area of expertise. For instance, one team deals with the user interface and others concentrate on back-end services. Various areas of development require specialized knowledge like automated and manual testing. Test automation is different as it depends on the user interface, cascading style sheets implementation or REST API. 

A QA manager builds the teams carefully, comprising of team members with different skillsets, such as a QA architect, UI testers, functional testers, automation testers, and performance testers. In addition to this, they define responsibilities and roles of team members, and define how to go about tasks. They have to guarantee that the development testers are appropriately equipped to test their team’s specific development work. 

3. Monitor quality metrics 

QA manager has to take a macroscopic view of quality by observing the trends in metrics across all teams. This provides a complete view of the quality code of the organization and in recognizing teams that may not be working up to the mark. 

There are many reasons why the teams might not be performing according to expectations. They may not be following the defined testing techniques, which could possibly heighten the number of defects. If the QA manager has the right metrics in place, she can easily get to the root of the issue, carefully drill down into the team’s work practices, and bring them up to par.

4. Observe system quality 

It’s the QA architect’s choice whether to carry out cross-functional testing to guarantee that everything performs well together. They test the connection between modules in various configurations and environments to guarantee that the system is functioning properly, without any performance, security, or integration issues. 

QA manager’s main responsibility is to reflect on the overall quality of their system. When they look at issues from a system level, they can think of themselves as a bridge between the development, testing, and QA teams. 

5. Be able to handle demands by themselves 

QA managers should be able to handle demands and arrange resources by themselves, rather than relying on external help such as more QA resources or tools or money. Simply put, they should be able to re-organize existing resources to meet the demands. As an agile unit, ideally all team members should have the competence to run various tests, as all members of the team have multiple skills and anyone can rise to the occasion.  

In certain situations, the QA manager may move testers between teams to address requirements for additional testing. This method is called “swarming.” 

If you are a QA manager who is seeking ways to thrive in an agile environment, be sure to keep the above points in mind–and watch yourself succeed!

Key Benefits of Test Case Repositories for QA Teams

This post was originally published on this site

If every team member maintains his or her own document version of test cases, then there is a high possibility for them to lose track of the entire test plan and its execution.

The result is patchy test coverage, missed execution of test cases, and poor product delivery. 

Keeping test cases in an organized way has big payoffs, especially in medium to large projects. 

Further, testing is a repeatable process. Re-using test cases saves time for everyone. Projects have large parts that can be repeated for testing. Maintaining a test case repository allows you to reuse previous test resources wherever needed and speed things up. The good news is maintaining an organized test case repository is not even difficult! 

Success of a software testing team is generally associated with the volume and nature of test cases that form the basis of testing cycles. It may require a tremendous amount of time and attention to the assimilation of test cases, with the primary focus locked on establishing a comprehensive test case repository for each application.

Test repository comprises test cases that encompass all key possibilities of permutations and combinations in workflow execution and transaction, ensuring that all variations in system and user interactions are covered.

A test case repository is not just confined to variations in system and user interactions but also caters to:

  • Different target configurations spanning form factors, databases, operating systems, hardware platforms.
  • Changes in software releases and versions that should be forward and backward compatible. 

Source: Harvard Business Review

Let’s have a closer look at why the test repository is a good first step and how it can be helpful for groups observing lean test management.

Overall waste reduction

The biggest value of lean methodologies in software testing is that they help organizations improve the reuse of available resources by mitigating waste. Experts believe that old testing processes are often focused on the documentation of each testing effort and running tests again at the end of each release–to ensure that new work didn’t impact the previous functionality. However, this approach is problematic as it requires writing and maintaining test cases demanding more costs, time, and effort. 

In a modern test management tool, a test repository crops up as you write test cases and keep associating with requirements. It is a live working document that is embedded in your project in a granular way. QA teams can write tests once and easily access them for future use. Moreover, testers can ensure seamless automation and execute tests as per requirements without needing manual approval for each interaction. This gives the QA professional enough time to focus on other critical areas and assure that the project is fully supported as required.

Moreover,  as a test case repository grows with time, testers should keep it current with every update to the business application or software product. If this is not done, then it loses synchronization with the actual function and behavior of the software over time. This consequently degrades the results produced in future QA cycles.

Below snapshot shows an example of a well-organized list of test cases. Each test case associated and grouped neatly in test scenarios and requirements. A good test repository app provides detailed dashboards such as this for quick tracking of previous test assets.

Boost application quality and knowledge

Test repository plays a vital role in ensuring that each system component is fully evaluated and hidden defects are identified quickly. Studies suggest that the key to achieving a successful and smooth project execution is possibly the maintenance of system documentation which includes a test repository. 

If the code does not pass the test case in the very first run, QA teams can leverage the repository to locate if certain areas of the software are more prone to errors than others, and then take a systemic approach to fix the whole branch. Testing efforts are reduced with a maintained central repository containing extensive data for all kinds of testing.

A unified platform will be less expensive and more accessible than one dedicated to each group or project, helping to boost overall quality and keep everyone on track.

Standardizing processes

We often see teams leveraging different procedures depending on the project but this may not be the most efficient way. Lean testing helps standardize processes and keeping a repository can further boost this initiative. Storing unwanted and excessive data can make retrieving appropriate information a challenging task. But with the help of lean practices, teams can easily refresh tests to reflect the most current trends.

Depending on the frequently used datasets, you can easily eliminate obsolete data to ensure the presence of new and quality data. This reduces the costs to store unnecessary data. Additionally, having different versions of the repository can greatly help in regression testing to identify what changes in data can break the code.

Source: Transforming Data With Analytics 

Significance of test case repository tools

Imagine the need of tracking your development and testing progress but having no singular source. In this regard, choosing the right test case repository tool that is versatile and performs according to the project requirements is perhaps very important.

Test repository tools, like offered by Kualitee, can help you leverage your lean testing in the following ways: 

  • Create an organic test case repository without additional effort for documentation.
  • Align your test cases according to your requirements and modules for ease of use.
  • Manage manual and automated test cases from one place.
  • Keep track of their execution records.
  • Check ownership of creation and editing.