A systematic literature study of regression test case prioritization approaches

Regression testing is about running the entire test ensemble again to ensure that amendments do not negatively affect the system. A popular approach in regression testing is test case prioritization which reorders test cases in such a way that those with higher priorities are run earlier than those with lower priorities based on some criterion. Numerous researchers have worked on different aspects of prioritization approaches. This paper presents the result of the study conducted on different prioritization approaches showing mostly stressed areas by researchers and the areas where there is a future scope. For it, studies related to test case prioritization in regression testing from the year 2004 to 2018 are analyzed by dividing this time period into three slots of five years each. 36 studies were selected from 948 studies to answer the research questions framed for this study. The trends followed in TCP along with the approaches evolved are thus documented to find the current trends and future scope for the researchers to work upon.


Introduction
"Regression testing is selective retesting of a system or component to verify that modifications have not caused undesirable effects and that the system or component still complies with its specified requirements" [1]. In this process, the retesting of a system is performed in which both existing and newly added test cases were executed [2]. The experiment conducted by an industry collaborator showed that more than 50 days are required to test 20,000 lines of code [3]. Hence researchers proposed diverse techniques to improve regression testing performance so as to make it a cost-effective process. In the study conducted by Yoo and Harman, these techniques are categorized as test suite minimization, test case selection, test case prioritization [4].

Test case minimization
This approach works towards eliminating the test cases, which becomes redundant with regard to coverage of some set of program requirement thereby decreasing the number of test cases in a regression test suite [5]. It states that only a subset of the test suite is economical to use. Minimization is sometimes called "test suite reduction" signifying that the deletion of test cases is everlasting [5].

Test case selection
These techniques select some of the test cases which focus on testing the changed part of the software. It does not remove test cases; rather it filters the ones which are related to the modified portion of the source code [6].

Test case prioritization
Prioritization approach states that as the execution of all the test cases is not practical due to resource constraints, therefore the test cases are ordered in a way such that those with higher priorities are run prior to those with lower priorities based on some criterion [7]. These techniques are usually preferred because it deals with the original test suite and no test cases are eliminated from the initial test suite [8].

Literature study
Numerous studies were performed by researchers and several prioritization methods were introduced, with an aim to enhance fault detection rate. These methods were categorized into several domains according to the nature of the study conducted by the researchers. Yoo and Harman conducted a survey in which various regression testing approaches were examined to enhance the significance of the assembled test suite in regression testing. The study performed by them is elaborated correspondingly into three spheres, that isminimization, selection and prioritization format. TCP approaches were discussed in detail along with their evaluation metrics. The most popular approaches as defined by them are Coverage based, History based, Cost-based, model-based and requirement based and other approaches [4]. Kumar and Singh performed a detailed overview of coverage and cost-based approach. They have elaborated that software development cost will eventually be reduced if the new techniques are implemented in the initial phases of the software development life cycle [9]. Kiran and Chandra Prakash have performed an overall review of regression testing for test case prioritization. They have also discussed several TCP techniques along with their strategies and algorithms developed so far. They have concluded that survey analysis eases the way of performing TCP. But, the benefits of the various parameters as well as cost factors should be taken care of. The software testing sums to nearly 50% of the whole cost of its development. If the path testing is carried out, then almost 65% of the errors in the software will be detected by itself [10]. Catal and Mishra, have focused on documenting the current trends of test case prioritization in their study and the trend of publication are shown with the types of dataset and metrics used [11]. Singh et al. carried out a systematic literature study for regression test case prioritization techniques for the time period of 1997 to 2011. The authors have discussed that in spite of various approaches followed by different techniques, the main aim of prioritizing test cases remains the same i.e. to increase fault detection rate. It is concluded that; as no broad techniques exist, there is a necessity to carry out experimental comparisons amongst the existing techniques and to derive out new and effective techniques [12]. Khatibsyarbini et al. performed a systematic literature review on TCP approaches in regression testing for the time period of 1999 to 2016. The authors have concluded that despite the fact the TCP approaches are considerably worked upon by numerous researchers in software testing industry, there are a number of prioritization techniques which have a scope for further improvement in the type of data used and thus the execution process followed by each approach [13]. Omdev et al. have performed a literature study of TCP approaches in which they concluded that various approaches can be combined or used as a hybrid so as to gain the advantage of each other in achieving better and remarkable results [14]. Solanki et al. have performed a comparative performance evaluation of modified ant colony optimization (m-aco) technique with other existing nature-inspired techniques. The results deduced by them have shown the superiority of m-aco in reaching optimal or near-optimal solutions [15]. Solanki et al. have performed an experimental analysis of m-aco technique for TCP with popular software testing problems and have shown the effectiveness of their approach using APFD and PTR Metrics [16]. This study was conducted with an aim to contribute towards the software testing domain by showing the trend which is going on in regression test case prioritization. For it, certain research questions are devised which will help us to focus on research papers which will show us the prevailing trend of publication of research papers in the regression test case prioritization domain.

Research method
This study aims to analyze the research work of numerous researchers in developing various test case prioritization approaches used in regression testing field. The guidelines of Kitchenham [17] were followed in presenting this study in a systematic and well-structured format. The given Fig. 1 shows the sequence of steps which were followed.  Figure 1 depicts the steps followed in documenting this study. In the first phase, research questions are formulated which answers the importance and need for this paper review. In the next phase, search criteria are defined so as to find the most relevant papers. After this phase, the most crucial phase is searching the data repository; this study selected Google Scholar for searching the papers as it comprises papers from all data repositories whether it may be IEEE Xplore, Springer, ACM digital library, and Elsevier. Next phase deals with selecting the most relevant papers after following the exclusion and inclusion criteria. After this step 36 studies were finalized which answers the research questions. Subsequently, data extraction is performed to draw some fruitful conclusion regarding current trends and future scope.

Research questions
The aim of deciding the research questions is to screen the papers on the basis of these questions. RQ

Search criteria
Initially, the phrase Test case prioritization was used as a search query in Google Scholar. The starting year was set to 2004, till 2018. It returned a huge amount of literature with number mounting to 71,600. It was practically not feasible to go through so many papers. So, the search string is modified as "Test case prioritization" with same starting year to till date (April 2018). 3435 number of papers were retrieved, which is still not feasible. So, the search results were narrowed by modifying the search string to "Test case prioritization" <and> "Regression testing" <and> "Software testing". The results obtained were 2132 in number. The search results were further narrowed and finally a search string is adopted, which is, "Test case prioritization" <and> "Regression testing" <and> "Software testing" <and> "Selection" <and> "Minimization". The results obtained were 948 in number.

Study selection
Of the results obtained by using the search string, selection of the relevant papers was performed based on the research questions and relevance to the topic i.e. which fulfills the validity of this systematic study.

Research papers selected for the time span 2009-2013
Yoo and Harman conducted a survey in which various regression testing approaches were examined for the significance of the assembled test suite in regression testing. The study performed by them is elaborated correspondingly into three spheres, that is -minimization, selection and prioritization format. TCP approaches were discussed in detail along with their evaluation metrics [4]. Arafeen

Research papers selected for the time span 2014-2018
Nardo et al. have conducted a case study on industry system of real regression faults using coverage-based regression testing techniques. The authors examined selection, minimization, prioritization and the combination of selection and minimization approaches and deduced their results [45]. Mayan   Hettiarachchi et al. [49] 2014 Requirements-risk based TCP.
Silva et al. [53] 2016 Hybrid method for test case selection and prioritization. 33.

Results and analysis
In this study, the existing literature from 2004-2018 was studied. The results will now be generalised from the selected papers by subdividing them according to their potential of answering the research questions.

What percentage of TCP techniques are focused/oriented towards capturing user requirements/faults/cost? (Research question 1)
Out of the selected studies, for the time slot of 2004-2008, 5 such papers were identified, [ [44], and out of these [44] was able to fulfill criteria of RQ 2 as well, therefore it was included for it also. For the time slot of 2014-2018, 8 such papers were identified, [47 -53] [55].
The analysis of these papers shows that the main criterion for the development of diverse techniques is to enhance the rate of fault detection. The main difference between them is the execution sequence followed and the metrics used. In the selected studies requirement-based approach is favored by most of the researchers followed by fault-based and then cost-effectiveness based. From this, we can conclude that exploration of this approach has started and therefore it was chosen by researchers to work on it, as seen by the choice of researchers in selecting this approach to work towards developing new and effective TCP approach.

What percentage of test case prioritization techniques developed so far are based on nature-inspired algorithms? (Research question 2)
Out of the selected studies, for the time slot of 2004-2008, 3 such papers were identified, [30 -32] [57] which have fulfilled the defined criteria of the research question. The reason stated by them for using nature-inspired algorithms is about their high efficiency in reaching the optimal solution to realworld problems by using successful characteristics of the biological system. From this, one can conclude that the usage of nature-inspired algorithms in developing TCP techniques has gained popularity in recent years. Earlier, it was an unexplored area, but now it was favoured by the researchers seeing the potential and effectiveness of these algorithms in solving problems in computer science.

Do some regression testing techniques available that supports hybrid of Selection/Minimization and Prioritization? (Research question 3)
Out of the selected studies, for the time slot of 2004-2008, 2 such papers were identified, [24] [29] which fulfills the defined criteria of the research question and therefore included in the study. For the time slot of 2009-2013, 4 such papers were identified, [34] [38] [39] [43] and out of these [34] [38] [39] were able to answer RQ 2 as well, therefore, it was also included there. For the time slot of 2014-2018, 3 such studies were identified, [45] [46] [54] which have answered the research question. From this, it can be deduced that every approach has its own advantages and limitations, if one is used in the combination or hybrid with the other, the researchers will be able to exploit the strengths of combined approaches as well as in overcoming their shortcomings so that effectiveness and potential of finding faults within limited time and cost can be achieved, subsequently benefitting the entire software testing community. Following is the year wise distribution of the selected studies for the respective years.

Research findings
During the testing phase, the prioritization of test cases is an essential activity [58]. A proper combination of test plan and test cases are necessary for the development of error-free quality software within planned budget and time. At a time when client hopes are high, effective software testing will ensure delivery of reliable and quality-oriented software product, more accurate results, lower maintenance costs and more satisfied users. However, ineffective testing will lead to the opposite results; low-quality products, inadequate results, unreliable product, higher maintenance costs, unhappy users. For this implementation of TCP technique is very beneficial so that faults may be detected at the earliest and changes can be made accordingly. Therefore, the role of test case prioritization in regression testing is much more emphasized. While performing the analysis of the studies included for answering the Research question 1, it can be concluded that though there are diverse approaches of TCP developed by numerous researchers with the time being, a shift in the trend was observed, which was noticed during the study selection phase. In the starting period search-based approaches were utilized the most, then coverage based, then fault-based, then history based and then requirement based, followed by other approaches. It is observed that while a lot of work has been done on most of the popular approaches, now requirement-based prioritization holds a scope to work upon. As a system is built on its requirements, so by capturing maximum user requirement and testing those functionalities first which are related to those requirements will lead to delivery of the reliable product.
Of the selected studies, 12 were based on requirement-based TCP which is approximately 33%. For research question 2, it is observed that nature-based algorithms were not so popular earlier, to solve problems in computer science.
A paradigm shift has been observed and various researchers have started using these techniques due to their high efficiency in reaching towards the optimal solution. Out of the selected studies, 13 have focused on the usage of nature-inspired algorithms to work on TCP techniques which are approximately 36%. Another inference which can be drawn from the analysis of the studies is that earlier only simple versions of nature-inspired techniques were in trends, such as ACO, BCO, and GA. The coming years has shown the trend of developing the modified versions of these techniques, as well as their usage in hybrid form. The reason for modifying these is to achieve efficient results in form of enhanced fault detection as compared to its previous versions. So, it can be said that nature-inspired techniques are also open to improvement. For research question 3, it is observed that though TCP techniques are favoured by researchers to work upon, they are also used as a hybrid with other regression testing techniques such as selection and minimization. Out of the selected studies, 9 were based on the usage of the hybrid approach. The reason to be concluded for the usage of the hybrid approach is to maximize as well as early fault detection so that improvements can be made. In these two or more approaches are used in such a conjunction that researchers were able to exploit the strengths of the combined approaches and their shortcomings were suppressed by each other. This has led the testing community to work upon it so that delivery of the fault-free and reliable product can be ensured.

Threats to validity
This study has some recognized limitations that may jeopardize its validity. Similar to previous reviews there are various potential threats which are-• There might be a possibility that the authors were not able to access all the studies related to TCP which have answered all the research questions. There could be two reasons for it. Firstly, the search string used is too narrow. Secondly, as the authors are well versed in the English language, therefore, they have selected studies which are in the English language only. So, some relevant non-English studies could have been missed though it cannot be avoided. • The probability of occurrence of the problem of overfitting cannot be ruled out while studies were selected for answering the particular research question, though cross-checking of studies was done so as to avoid inclusion of inaccurate studies.

Conclusion
Test case prioritization (TCP) is one of the most popular approaches in regression testing. Diverse techniques of TCP have been proposed by numerous researchers. This study has worked to analyze the approaches based on the research questions framed. This study was performed by searching, classifying, analyzing and deducing outcomes from the selected studies. The motive behind this study is to find which of the approach, area or field holds a potential scope of improvement or holds future scope to work upon. From the conducted study, it can be deduced that the current research trend lies in developing a requirement-based TCP technique using the natureinspired algorithm. This study has defined a base for future work by providing a deep insight to the testing community by exposing the latest trends. This will benefit the potential researchers in developing new techniques to find loopholes in the software system so that a time-bound, within budget, a reliable and robust product can be delivered to the users.