By Joke Depraetere, Christophe Vandeviver, Ines Keygnaert & Tom Vander Beken
Reviewing qualitative and quantitative research? Or aiming to develop a new theory based on literature readings? The relatively new review type, the Critical Interpretive Synthesis (CIS), allows for both. Emphasizing flexibility and a critical orientation in its approach, the CIS aims to develop a new coherent theoretical framework from both qualitative and quantitative research. Recognized as one of the best review types, the CIS provides a fresh interpretation of the data rather than a summary of results, as is often the case with other review types. However, CIS’ greatest advantage, flexibility is also one of the greatest disadvantages since it hampers its implementation, introduces ambiguity in its execution and reporting and therefore exacerbate concerns about trustworthiness.
In our published work in the International Journal of Social Research Methodology, evaluation criteria for the CIS were developed and applied on 77 published CIS reviews. By developing these criteria and assessing existing CIS reviews we aimed to evaluate the trustworthiness of these reviews and provide guidelines to future authors, journal editors and reviewers in their implementation and evaluation of the CIS.
The paper outlines two important concepts of trustworthiness in scientific research: transparency and systematicity. While transparency focusses on the reproducibility of the review process, systematicity emphasizes that fit-for-purpose methods need to be implemented and well executed. Previous scholars (Templier & Paré, 2017; Paré et al., 2016) have already developed various guidelines regarding transparency and systematicity in review types. They, however, remained broad and lacked a focus on the specificities that accompany these various review types. Each review type is characterized by different key features that allow to distinguish review types. These features should be transparently reported and soundly executed (i.e. systematicity). Some features can be considered as more central and important than other more peripheral features. This allows to identify a hierarchy of features and enables the evaluation of the extent to which central features of the review type have been consistently implemented and clearly reported in research.
Overall, seven key features are formulated and presented in a hierarchy based on the main goals of the CIS as emphasized by previous scholars (Dixon-Woods et al., 2006b; Entwistle et al., 2012). Both aspects of trustworthiness were evaluated, allowing us to make a distinction between transparency and systematicity of the various key features. During our evaluation of the CIS reviews, we identified six groups of papers based on the scoring of these key features. While only 28 papers transparently reported and soundly executed the four highest ranked features in the hierarchy, the majority of the papers (i.e. N = 47) did well on the two most important features of the CIS. These most important features represent the main goal of the CIS, namely the development of a theoretical framework using methods as described by the original authors of the CIS (Dixon-Woods et al., 2006). This, however, indicated that over 38% of the papers cannot be considered as trustworthy in terms of transparently reporting and soundly executing the two highest ranked features of the CIS.
The paper details which key features of the CIS were soundly executed and transparently reported and which features performed rather poorly. We conclude how the trustworthiness of CIS papers could be improved by providing various recommendations for future scholars, reviewers and journal editors regarding the implementation and evaluation of CIS reviews. While this paper only focuses on one review type, we hope that this paper may be considered as a starting point for developing similar evaluation criteria for methodological reporting in other review genres.
To read the full IJSRM here.