The Importance of Regression Testing

July 18, 2022

Blog

The introduction of the Continuous Integration/Continuous Deployment (CI/CD) process has strengthened the release mechanism, helping products get to market faster than ever and allowing application development teams to deliver code changes more frequently and reliably.

Regression testing is the process of ensuring that no new mistakes have been introduced in the software after the adjustments have been made by testing the modified sections of the code as well as the parts that may be affected by the modifications.

The Software Testing Market size is projected to reach $40 billion in 2020 with a 7% growth rate by 2027. Regression testing accounted for more than 8.5 percent of market share and is expected to rise at an annual pace of over 8% through 2027 as per the reports stated by the Global Market Insights group.

Regression testing is a must for large-sized software development teams following an agile model. When many developers are making multiple commits frequently, regression testing is required to identify any unexpected outcome in overall functionality caused by each commit. CI/CD setup identifies that and notifies the developers as soon as the failure occurs and makes sure the faulty commit doesn’t get shipped into the deployment.

There are different CI/CD tools available, but Jenkins is widely accepted because it is open source, hosts multiple productivity improvement plugins, has active community support, and can set up and scale easily. Source Code Management (SCM) platforms like GitLab and GitHub also provide a list of CI/CD features and are preferred when using a single platform to manage code collaboration along with CI/CD. 

With the increasing complexity, it is important to have an effective notification mechanism, robust monitoring, balanced load distribution of clusters, scalability and maintenance support, and priory management. In such scenarios, the QA team can focus on CI/CD optimization, shortening time to market, and achieving the committed release timeline.

Regression Testing in Continuous Integration / Delivery (Source: Softnautics)

Effective Notification Mechanism

CI/CD tools like Jenkins provide plugin support to notify team members of unexpected failures during the regression testing. But when these notifications increase, it becomes inefficient to investigate each of them, leading to a greater probability of oversight. To avoid such scenarios, a Failure Summary Report (FSR) highlighting new failures can be generated. FSRs may include an executive summary section along with detailed summary sections.

Based on the project requirements, one can integrate JIRA, Jenkins links, SCM commit links, and time stamps into the FSR, which can be generated as needed.

Optimum Use of Computing Resources

When CI/CD pipelines are set up to use a cluster of multiple hosts with high computing resources, it is expected to have a minimum turnaround time of a regression run cycle with maximum throughput. To achieve this, regression runs need to be distributed correctly across the cluster. Workload management and scheduler tools like IBM LSF and PBS can be used to run jobs concurrently based on available computing resources at a given point in time.

In Jenkins, one can add multiple slave nodes to distribute jobs across the cluster to minimize the waiting time in the Jenkins queue, but this needs to be executed based on available computing power after understanding the resource configuration of slave hosting servers. If not done carefully, it can result in node crash and loss of data. 

Resource Monitoring

To support the growing requirement of CI/CD, while scaling one should consider the disk space limitations or cluster resource limitations. If not handled properly, it results in CI/CD node crashes, slow executions, and loss of data. If such an incident happens when a team is approaching an import deliverable, it becomes difficult to meet the committed release timeline. Such scenarios can be avoided with a robust monitoring and notification mechanism. You can also build a monitoring application that continuously watches the resources of each computing host, network disk space, and local disk space and raises a red flag when the set thresholds are crossed.

Scalability and Maintenance

When regression job count grows to over a thousand, it can be challenging to maintain. Manually implementing a single change over so many jobs is time-consuming and increases potential for error. To overcome this challenge, one should opt for a modular and scalable approach while designing test procedure run scripts. Instead of writing steps in CI/CD, one can opt to use SCM to maintain test run scripts or use Jenkins APIs to update the jobs from the backend to save manual efforts.

Priority Management

When regression testing of multiple software products is being handled in a single CI/CD setup, priority management becomes important. Pre-merge jobs should be prioritized over post-merge jobs. This can be achieved by running pre-merge jobs on a dedicated host, providing separate Jenkins slave and LSF queue. Post-merge Jenkins jobs of different products should be configured to use easy-to-update placeholders for Jenkins slave tags and LSF queues such that priorities can be easily altered based on which product is approaching release.

Integration With Third-Party Tools

When using multiple SCMs like GitLab and GitHub alongside issue tracking tools like JIRA, tacking commits, MRs, PRs, and issue updates help the team stay in sync. Jenkins integration with GitLab and GitHub helps in reflecting pre-merge run results into SCM. By integrating an issue tracker like JIRA with Jenkins, one can create and update issues based on run results. With SCM tools and JIRA integration, issues can be auto-updated on new commits and PR merges.

Not only must regression test plans be updated to reflect new changes in the application code, but they must also be iteratively improved to become more effective, thorough, and efficient. A test plan should be viewed as an ever-evolving document. Regression testing is critical for ensuring high quality, especially as the breadth of the regression develops later in the development process. That's why prioritization and automation of test cases are critical in Agile initiatives.