featured, Notebook

Developing assessment criteria of trustworthiness for the Critical Interpretive Synthesis

By Joke Depraetere, Christophe Vandeviver, Ines Keygnaert & Tom Vander Beken

Reviewing qualitative and quantitative research? Or aiming to develop a new theory based on literature readings? The relatively new review type, the Critical Interpretive Synthesis (CIS), allows for both. Emphasizing flexibility and a critical orientation in its approach, the CIS aims to develop a new coherent theoretical framework from both qualitative and quantitative research. Recognized as one of the best review types, the CIS provides a fresh interpretation of the data rather than a summary of results, as is often the case with other review types. However, CIS’ greatest advantage, flexibility is also one of the greatest disadvantages since it hampers its implementation, introduces ambiguity in its execution and reporting and therefore exacerbate concerns about trustworthiness.

In our published work in the International Journal of Social Research Methodology, evaluation criteria for the CIS were developed and applied on 77 published CIS reviews. By developing these criteria and assessing existing CIS reviews we aimed to evaluate the trustworthiness of these reviews and provide guidelines to future authors, journal editors and reviewers in their implementation and evaluation of the CIS.

The paper outlines two important concepts of trustworthiness in scientific research: transparency and systematicity. While transparency focusses on the reproducibility of the review process, systematicity emphasizes that fit-for-purpose methods need to be implemented and well executed. Previous scholars (Templier & Paré, 2017; Paré et al., 2016) have already developed various guidelines regarding transparency and systematicity in review types. They, however, remained broad and lacked a focus on the specificities that accompany these various review types. Each review type is characterized by different key features that allow to distinguish review types. These features should be transparently reported and soundly executed (i.e. systematicity). Some features can be considered as more central and important than other more peripheral features. This allows to identify a hierarchy of features and enables the evaluation of the extent to which central features of the review type have been consistently implemented and clearly reported in research.

Overall, seven key features are formulated and presented in a hierarchy based on the main goals of the CIS as emphasized by previous scholars (Dixon-Woods et al., 2006b; Entwistle et al., 2012). Both aspects of trustworthiness were evaluated, allowing us to make a distinction between transparency and systematicity of the various key features. During our evaluation of the CIS reviews, we identified six groups of papers based on the scoring of these key features. While only 28 papers transparently reported and soundly executed the four highest ranked features in the hierarchy, the majority of the papers (i.e. N = 47) did well on the two most important features of the CIS. These most important features represent the main goal of the CIS, namely the development of a theoretical framework using methods as described by the original authors of the CIS (Dixon-Woods et al., 2006). This, however, indicated that over 38% of the papers cannot be considered as trustworthy in terms of transparently reporting and soundly executing the two highest ranked features of the CIS.

The paper details which key features of the CIS were soundly executed and transparently reported and which features performed rather poorly. We conclude how the trustworthiness of CIS papers could be improved by providing various recommendations for future scholars, reviewers and journal editors regarding the implementation and evaluation of CIS reviews. While this paper only focuses on one review type, we hope that this paper may be considered as a starting point for developing similar evaluation criteria for methodological reporting in other review genres.

To read the full IJSRM here.

featured, Notebook

Radical critique of interviews – an exchange of views on form and content

By Rosalind Edwards (IJSRM Co-editor)

The ‘radical critique’ of interviews is a broad term encompassing a range of differing positions, but a shared element is an argument that interviews are not a method of grasping the unmediated experiences of research participants – that is, the content of the interview data.  Rather, the enactment of the method, of interviewer and interviewee exchanges, is data – that is, the form.  The critique has been the subject of a scholarly exchange of views in the Journal, drawing attention to agreements and distinctions in debates about radical critiques of interview data in social research.

In a themed section of the Journal on ‘Making the case for qualitative interviews’, Jason Hughes, Kahryn Hughes, Grace Sykes and Katy Wright contributed an article arguing that the focus on interviews as narrative performance (form) leaves in place a seemingly unbridgeable divide between the experienced and the expressed, and a related conflation of what can be said in interviews with what interviews can be used to say.  They call for attention to the ways that interview data may be used to discuss the social world beyond the interview encounter (content).

Jason Hughes, Kahryn Hughes, Grace Sykes and Katy Wright – ‘Beyond performative talk: critical observations on the radical critique of reading interview data’.

Emilie Whitaker and Paul Atkinson, responded to their observations, to argue that while their work (cited in Hughes et al.) urges methodologically-informed, reflexive analytic attention to interviews as speech events and social encounters (form), this is not at the expense of attention to content.  Indeed, they say, there cannot be content without form. 

Emilie Whitaker and Paul Atkinson – ‘Response to Hughes, Hughes, Sykes and Wright.

In reply, Hughes and colleagues state their intention to urge a synthesis that prioritises a focus on the content of interviews and the possibilities for what researchers can do with it, just as much as a critical attention to its form.

Jason Hughes, Kahryn Hughes, Grace Sykes and Katy Wright – ‘Response to Whitaker and Atkinson’.

The renditions of these constructive exchanges are my own, and may not (entirely) reflect those of the authors.

featured, Notebook

I Say, They Say: Effects of Providing Examples in a Survey Question

By Eva Aizpurua, Ki H. Park, E. O. Heiden & Mary E. Losch

One of the first things that survey researchers learn is that questionnaire design decisions are anything but trivial. The order of the questions, the number of response options, and the labels used to describe them can all influence survey responses. In this Research Note, we turn our attention to the use of examples, a common component of survey questions. Examples are intended to help respondents, providing them with information about the type of answers expected and reminding them of responses that might otherwise go unnoticed. For instance, the 2020 U.S. National Health Interview Survey asked about the use of over-the-counter medication, and included “aspirin, Tylenol, Advil, or Aleve” in the question stem. There are many other examples in both national and international surveys. Despite the potential benefits of using examples, there is a risk that respondents will focus too much on them, at the expense of overlooking cases not listed as examples. This phenomenon, called the “focusing hypothesis”, is what we test in our study.

Using an experimental design, we examined the effects of providing examples in a question about multitasking (“During the time we have been on the phone, in what other activities, if any, were you engaged [random group statement here]?”). In this experiment, respondents were randomly assigned to one of three conditions: the first group received one set of examples (watching TV or watching kids), the second group received a different set of examples (walking or talking with someone else), while the final group received no examples. Our goal was to determine whether respondents were more likely to report an activity (e.g., watching TV or walking) when it was listed as an example. We also wanted to understand whether providing examples resulted in respondents listing more activities beyond the examples.

We embedded this experiment in a telephone survey conducted in a Midwestern U.S. state and found support for the focusing hypothesis. As anticipated, respondents were more likely to mention the activity if it was provided to them as an example. However, the effect sizes were generally small and examples did not have an effect on the percentage of respondents who identified themselves as multitaskers, nor on the number of activities reported by them. This is because people faced with the experimental conditions were more likely to list the examples presented to them (i.e., watching TV, watching kids, walking, talking with someone else), while those in the control group more frequently reported activities outside this range (cooking, doing housework…), yielding no differences on the frequency of multitasking or on the number of multitasking activities.  Although examples can help respondents understand the scope of the question and remind them of certain responses, the results from this study indicate that they can also restrict the memory search to the examples provided. This has implications for survey practice, suggesting that the inclusion of examples in questions should be carefully considered and limited to certain situations, such as questions in which recall errors are anticipated or when the scope of the question might be unclear.

To learn more, see full IJSRM article here.

covid-19, Notebook

Adapting research with men during COVID-19: Experiences shifting to mobile phone-based methods

By Joe Strong, Samuel Nii Lante Lamptey, Richard Nii Kwartei Owoo, and Nii Kwartelai Quartey

It is impossible to understand masculinities without social research methods. Speaking and interacting with men is the fundamental cornerstone of the project Exploring the relationships between men, masculinities and post-coital pregnancy avoidance. Conducting these methods through ‘non-social’, distanced means, as a response to COVID-19, presents new challenges and opportunities and ethical considerations.

The original research sample frame was men aged 16 and over, who slept (proxy for ‘resident’) for at least some of their time in the study area. The research team were predominantly based / resident in the study area [a suburb of Accra], and all were living in Ghana prior to the declaration of a pandemic in 11 March 2020.

Response to COVID-19

The original research design necessitated close contact between respondents and the research team, using a household survey, focus group discussions and in-depth interviews. This proximity was quickly deemed unacceptable when compared to public health best practice (social distancing, limited movement, etc). Such methods endanger the respondents and the research team.

As it became evident that the pandemic was long-term, the team discussed potential mechanisms through which to continue the research in a safe and responsible manner. Mobile phone technology emerged as the only feasible way to ensure that social distancing and limited movement would be required for the research project to continue.

In the study area, mobile phone use is relatively high, reflecting broader trends in Ghana. However, these mobile phones were not all ‘smart’, i.e., it could not be assumed that respondents would have access to data or internet on their mobile devices. As such, continuing person-to-person survey interviews was the most feasible way, so as not to limit the sample to a) access to smart technology and b) ability / desire to navigate an online survey.

Thus, focus group discussions were removed entirely from the research design, as these could not be facilitated meaningfully through non-smart mobile phones. The survey questionnaires and in-depth interview schedule could remain the same, with additional questions on the impact of COVID-19. These had been tested prior to the pandemic in person to check for consistency, comprehension and relevance.

Practicalities

Obtaining equipment for the team in a timely and safe manner was essential – this included a mobile phone and three sim cards for each of the major telecommunication networks in the area. Fortunately, the team each had smart phone technology that allowed for communications to continue over WhatsApp.

Ethical amendments were submitted to account for consent being provided verbally, as written consent required inappropriate close contact. A huge outcome of the ethical amendment was the removal of anyone who could not consent for themselves. This has serious implications for the inclusivity of and representativeness of this research. The nature of the gatekeeping could not be observed or accounted for over the mobile phone. For example, it would not be clear if the parent of an adolescent – who required parental consent – would be in the room listening in. Critical voices, such as adolescents, people who need assistance with communication, e.g. sign language interpreters, are also not able to be incorporated into the survey.

The household listing conducted prior to the pandemic did not collect mobile phone information, as retrieving mobile numbers for each household member would be cumbersome and invasive. Thus, no sampling frame was available for the survey. To mitigate this, the study uses respondent driven sampling, whereby each survey respondent is asked to recruit three people from their personal network to be surveyed next and is compensated per successful recruit as well as for their own survey.

The experience of new methods

The use of mobile phones allows the respondents to decide when and where they want to be surveyed, providing them with greater autonomy than a household survey. In many ways, it empowers the respondent to have much more control over the survey. However, this also can make it harder, as the lack of physical presence makes distraction / missing a call much easier.

Moreover, the element of “out of sight, out of mind” hinders the efficiency with which respondents might recruit their friends, and the additional effort of conducting this recruitment through mobile phones might not help. We created regulations – no calling the same person twice in one day if they picked up, no more than three times in one week, end contact if asked – to try and mitigate overburdening respondents with reminders that might feel harassing.

We are finding that some respondents are reticent to be interviewed over the phone, preferring face-to-face interviews so that they might see the interviewer and build trust through sight. Despite the easing of lockdown in Ghana on 20 April 2020, the decision was made to maintain strict protocols of distancing between data collectors and respondents. This reflects the causes behind the ease of lockdown and that our research is non-essential, and we have a duty to avoid risking ourselves and the respondents. 

Responses to the lack of face-to-face cues were mixed. It makes it harder to use e.g. body language to gauge the respondent experience of the survey. On the other hand, it preserves a greater sense of anonymity for the respondent. It is necessary that data collectors “check-in” on respondents during an interview to ensure that the interview questions are not causing undue harm or stress, and that respondents be reminded that they are in control of the interviews. It is important that we acknowledge that the mobile phone becomes a part of the “context” of the research and it is essential to reflect on the impact of this.

Such experiences provide important opportunities for learning. Generally, we are finding that men are not afraid to talk to us over the phone. But we must acknowledge how many more men will be excluded through these methods and consider opportunities for their future inclusion. The greater control respondents have in arranging interviews to suit themselves is an important reminder of the need for patience and respect for respondents’ priorities and the (non-)essentialness of research.

At the time of writing (30 July 2020), 73 respondents have completed interviews, not including 22 seeds. For ongoing data visualisations and sneak peaks, visit the project website at: https://www.masculinitiesproject.org/

covid-19, featured, Notebook

Are novel research projects ethical during a global pandemic?

By Emily-Marie Pacheco and Mustafa Zaimağaoğlu

The global pandemic has inspired a plethora of new research projects in the social sciences; scholars are eager to identify and document the many challenges the COVID-19 situation has introduced into our daily lives, and explore the ways in which our societies have been able to thrive during these ‘unprecedented times’. Given the wide acknowledgement that life during a global pandemic is often more difficult than in our pre-pandemic circumstances, researchers must consider whether asking those in our communities to donate their time and energy to participating in our research is acceptable. Does recruitment for research which seeks to explore the psychological wellbeing and adjustment of those living through uniquely challenging circumstances during COVID-19 really reflect research integrity?

There is no simple answer to whether asking people to share their stories and experiences of COVID-19 is ethical or improper. Many would argue that social research has the potential to contribute many vital insights about life during a global pandemic which are unique to the humanistic lens and approach often reserved for the social sciences; such investigations could propel scholarly dialogue and manifest practically in recommendations for building resilient societies. However, social scientists have a responsibility to protect their participants from any undue harm they may experience as a result of their participation in a study. Thus, while social research may be especially important during a global pandemic, traditional study designs need to adapt to the circumstances of the pandemic and be held to higher ethical expectations by governing bodies and institutions.

Ethical social research during a global pandemic is reflected in research methods which demonstrate an awareness that we are asking more of our participants than ever before. Simple adaptations to existing projects can go a long way in bettering the experience of participants, such as by providing prospective participants additional information on what is expected of them if they choose to participate in a study – whether it be an online survey or an interview. Projects which aim to collect data using qualitative or interpersonal methods should be especially open to adaptation. These studies may be more ethically conducted by offering socially distant options, such as online focus groups or telephone interviews; adopting multimethod approaches and allowing participants the opportunity to contribute to projects in a medium which is most suitable for them may also be an ideal approach, such as by allowing participants the option to participate in online interviews or submitting audio-diaries conducted at their own discretion.

Attention should also be given to the various details of the research design which pertain to participant involvement more specifically. Does that online survey really needto include fifteen scales, and do they really need to ask all thosedemographic questions? Do online interviews really need to exceed thirty minutes and is it really necessary to require participants to turn their cameras on (essentially inviting you into their homes)? The ‘standard procedures’ for collecting data should be critically re-evaluated by researchers in consideration of the real-world context of those from whom they wish to collect data, with the aim of upholding their commitment to responsible research practices. Ethics boards should also aid researchers in identifying areas of their research designs which may be adapted to protect participants. This additional critical perspective may highlight participation conditions that may be arduous for participants, but which may have been overlooked as part of a traditional research design. 

Research during unprecedented times should also aim to provide a benefit to participants who generously donate their time and energy despite experiencing various transitions and changes in their own personal lives. While some researchers may need to devise creative solutions to meet this aim, many research methods in the social sciences have the inherent potential to serve as an activity which provides a benefit to those who engage in their process. For example, researchers may opt to collect data through methods which have a documented potential for promoting psychological wellbeing, or which are also considered therapeutic mechanism. Such approaches include methods which ask participants to reflect on their own experiences (e.g., audio-diaries, reflective entries, interviews with photo-elicitation) and those which focus on positive thoughts or emotions (e.g., topics related to hope, resilience, progress). Beyond these recommendations, researchers should also consider whether they really need participants at all. There are many options for conducting valuable research with minimal or no contact with participants, such as observational methods, content analyses, meta analyses, or secondary analyses. Some may argue that research during a global pandemic should only be conducted with either previously acquired or secondary data; others may argue that primary data collected voluntarily from willing participants is entirely ethical. Either way, respecting participants and their role in our research is always necessary. Beyond the requirements of doing so to uphold institutional research integrity expectations, it is our individual responsibility to ensure we, as researchers, are protecting those who make our work possible by assessing vulnerability, minimizing risk, and enhancing benefit, of participation – to the full extent of our capabilities.