featured, Notebook

Developing assessment criteria of trustworthiness for the Critical Interpretive Synthesis

By Joke Depraetere, Christophe Vandeviver, Ines Keygnaert & Tom Vander Beken

Reviewing qualitative and quantitative research? Or aiming to develop a new theory based on literature readings? The relatively new review type, the Critical Interpretive Synthesis (CIS), allows for both. Emphasizing flexibility and a critical orientation in its approach, the CIS aims to develop a new coherent theoretical framework from both qualitative and quantitative research. Recognized as one of the best review types, the CIS provides a fresh interpretation of the data rather than a summary of results, as is often the case with other review types. However, CIS’ greatest advantage, flexibility is also one of the greatest disadvantages since it hampers its implementation, introduces ambiguity in its execution and reporting and therefore exacerbate concerns about trustworthiness.

In our published work in the International Journal of Social Research Methodology, evaluation criteria for the CIS were developed and applied on 77 published CIS reviews. By developing these criteria and assessing existing CIS reviews we aimed to evaluate the trustworthiness of these reviews and provide guidelines to future authors, journal editors and reviewers in their implementation and evaluation of the CIS.

The paper outlines two important concepts of trustworthiness in scientific research: transparency and systematicity. While transparency focusses on the reproducibility of the review process, systematicity emphasizes that fit-for-purpose methods need to be implemented and well executed. Previous scholars (Templier & Paré, 2017; Paré et al., 2016) have already developed various guidelines regarding transparency and systematicity in review types. They, however, remained broad and lacked a focus on the specificities that accompany these various review types. Each review type is characterized by different key features that allow to distinguish review types. These features should be transparently reported and soundly executed (i.e. systematicity). Some features can be considered as more central and important than other more peripheral features. This allows to identify a hierarchy of features and enables the evaluation of the extent to which central features of the review type have been consistently implemented and clearly reported in research.

Overall, seven key features are formulated and presented in a hierarchy based on the main goals of the CIS as emphasized by previous scholars (Dixon-Woods et al., 2006b; Entwistle et al., 2012). Both aspects of trustworthiness were evaluated, allowing us to make a distinction between transparency and systematicity of the various key features. During our evaluation of the CIS reviews, we identified six groups of papers based on the scoring of these key features. While only 28 papers transparently reported and soundly executed the four highest ranked features in the hierarchy, the majority of the papers (i.e. N = 47) did well on the two most important features of the CIS. These most important features represent the main goal of the CIS, namely the development of a theoretical framework using methods as described by the original authors of the CIS (Dixon-Woods et al., 2006). This, however, indicated that over 38% of the papers cannot be considered as trustworthy in terms of transparently reporting and soundly executing the two highest ranked features of the CIS.

The paper details which key features of the CIS were soundly executed and transparently reported and which features performed rather poorly. We conclude how the trustworthiness of CIS papers could be improved by providing various recommendations for future scholars, reviewers and journal editors regarding the implementation and evaluation of CIS reviews. While this paper only focuses on one review type, we hope that this paper may be considered as a starting point for developing similar evaluation criteria for methodological reporting in other review genres.

To read the full IJSRM here.

featured, Notebook

Radical critique of interviews – an exchange of views on form and content

By Rosalind Edwards (IJSRM Co-editor)

The ‘radical critique’ of interviews is a broad term encompassing a range of differing positions, but a shared element is an argument that interviews are not a method of grasping the unmediated experiences of research participants – that is, the content of the interview data.  Rather, the enactment of the method, of interviewer and interviewee exchanges, is data – that is, the form.  The critique has been the subject of a scholarly exchange of views in the Journal, drawing attention to agreements and distinctions in debates about radical critiques of interview data in social research.

In a themed section of the Journal on ‘Making the case for qualitative interviews’, Jason Hughes, Kahryn Hughes, Grace Sykes and Katy Wright contributed an article arguing that the focus on interviews as narrative performance (form) leaves in place a seemingly unbridgeable divide between the experienced and the expressed, and a related conflation of what can be said in interviews with what interviews can be used to say.  They call for attention to the ways that interview data may be used to discuss the social world beyond the interview encounter (content).

Jason Hughes, Kahryn Hughes, Grace Sykes and Katy Wright – ‘Beyond performative talk: critical observations on the radical critique of reading interview data’.

Emilie Whitaker and Paul Atkinson, responded to their observations, to argue that while their work (cited in Hughes et al.) urges methodologically-informed, reflexive analytic attention to interviews as speech events and social encounters (form), this is not at the expense of attention to content.  Indeed, they say, there cannot be content without form. 

Emilie Whitaker and Paul Atkinson – ‘Response to Hughes, Hughes, Sykes and Wright.

In reply, Hughes and colleagues state their intention to urge a synthesis that prioritises a focus on the content of interviews and the possibilities for what researchers can do with it, just as much as a critical attention to its form.

Jason Hughes, Kahryn Hughes, Grace Sykes and Katy Wright – ‘Response to Whitaker and Atkinson’.

The renditions of these constructive exchanges are my own, and may not (entirely) reflect those of the authors.

featured, Notebook

I Say, They Say: Effects of Providing Examples in a Survey Question

By Eva Aizpurua, Ki H. Park, E. O. Heiden & Mary E. Losch

One of the first things that survey researchers learn is that questionnaire design decisions are anything but trivial. The order of the questions, the number of response options, and the labels used to describe them can all influence survey responses. In this Research Note, we turn our attention to the use of examples, a common component of survey questions. Examples are intended to help respondents, providing them with information about the type of answers expected and reminding them of responses that might otherwise go unnoticed. For instance, the 2020 U.S. National Health Interview Survey asked about the use of over-the-counter medication, and included “aspirin, Tylenol, Advil, or Aleve” in the question stem. There are many other examples in both national and international surveys. Despite the potential benefits of using examples, there is a risk that respondents will focus too much on them, at the expense of overlooking cases not listed as examples. This phenomenon, called the “focusing hypothesis”, is what we test in our study.

Using an experimental design, we examined the effects of providing examples in a question about multitasking (“During the time we have been on the phone, in what other activities, if any, were you engaged [random group statement here]?”). In this experiment, respondents were randomly assigned to one of three conditions: the first group received one set of examples (watching TV or watching kids), the second group received a different set of examples (walking or talking with someone else), while the final group received no examples. Our goal was to determine whether respondents were more likely to report an activity (e.g., watching TV or walking) when it was listed as an example. We also wanted to understand whether providing examples resulted in respondents listing more activities beyond the examples.

We embedded this experiment in a telephone survey conducted in a Midwestern U.S. state and found support for the focusing hypothesis. As anticipated, respondents were more likely to mention the activity if it was provided to them as an example. However, the effect sizes were generally small and examples did not have an effect on the percentage of respondents who identified themselves as multitaskers, nor on the number of activities reported by them. This is because people faced with the experimental conditions were more likely to list the examples presented to them (i.e., watching TV, watching kids, walking, talking with someone else), while those in the control group more frequently reported activities outside this range (cooking, doing housework…), yielding no differences on the frequency of multitasking or on the number of multitasking activities.  Although examples can help respondents understand the scope of the question and remind them of certain responses, the results from this study indicate that they can also restrict the memory search to the examples provided. This has implications for survey practice, suggesting that the inclusion of examples in questions should be carefully considered and limited to certain situations, such as questions in which recall errors are anticipated or when the scope of the question might be unclear.

To learn more, see full IJSRM article here.

covid-19, featured, Notebook

Are novel research projects ethical during a global pandemic?

By Emily-Marie Pacheco and Mustafa Zaimağaoğlu

The global pandemic has inspired a plethora of new research projects in the social sciences; scholars are eager to identify and document the many challenges the COVID-19 situation has introduced into our daily lives, and explore the ways in which our societies have been able to thrive during these ‘unprecedented times’. Given the wide acknowledgement that life during a global pandemic is often more difficult than in our pre-pandemic circumstances, researchers must consider whether asking those in our communities to donate their time and energy to participating in our research is acceptable. Does recruitment for research which seeks to explore the psychological wellbeing and adjustment of those living through uniquely challenging circumstances during COVID-19 really reflect research integrity?

There is no simple answer to whether asking people to share their stories and experiences of COVID-19 is ethical or improper. Many would argue that social research has the potential to contribute many vital insights about life during a global pandemic which are unique to the humanistic lens and approach often reserved for the social sciences; such investigations could propel scholarly dialogue and manifest practically in recommendations for building resilient societies. However, social scientists have a responsibility to protect their participants from any undue harm they may experience as a result of their participation in a study. Thus, while social research may be especially important during a global pandemic, traditional study designs need to adapt to the circumstances of the pandemic and be held to higher ethical expectations by governing bodies and institutions.

Ethical social research during a global pandemic is reflected in research methods which demonstrate an awareness that we are asking more of our participants than ever before. Simple adaptations to existing projects can go a long way in bettering the experience of participants, such as by providing prospective participants additional information on what is expected of them if they choose to participate in a study – whether it be an online survey or an interview. Projects which aim to collect data using qualitative or interpersonal methods should be especially open to adaptation. These studies may be more ethically conducted by offering socially distant options, such as online focus groups or telephone interviews; adopting multimethod approaches and allowing participants the opportunity to contribute to projects in a medium which is most suitable for them may also be an ideal approach, such as by allowing participants the option to participate in online interviews or submitting audio-diaries conducted at their own discretion.

Attention should also be given to the various details of the research design which pertain to participant involvement more specifically. Does that online survey really needto include fifteen scales, and do they really need to ask all thosedemographic questions? Do online interviews really need to exceed thirty minutes and is it really necessary to require participants to turn their cameras on (essentially inviting you into their homes)? The ‘standard procedures’ for collecting data should be critically re-evaluated by researchers in consideration of the real-world context of those from whom they wish to collect data, with the aim of upholding their commitment to responsible research practices. Ethics boards should also aid researchers in identifying areas of their research designs which may be adapted to protect participants. This additional critical perspective may highlight participation conditions that may be arduous for participants, but which may have been overlooked as part of a traditional research design. 

Research during unprecedented times should also aim to provide a benefit to participants who generously donate their time and energy despite experiencing various transitions and changes in their own personal lives. While some researchers may need to devise creative solutions to meet this aim, many research methods in the social sciences have the inherent potential to serve as an activity which provides a benefit to those who engage in their process. For example, researchers may opt to collect data through methods which have a documented potential for promoting psychological wellbeing, or which are also considered therapeutic mechanism. Such approaches include methods which ask participants to reflect on their own experiences (e.g., audio-diaries, reflective entries, interviews with photo-elicitation) and those which focus on positive thoughts or emotions (e.g., topics related to hope, resilience, progress). Beyond these recommendations, researchers should also consider whether they really need participants at all. There are many options for conducting valuable research with minimal or no contact with participants, such as observational methods, content analyses, meta analyses, or secondary analyses. Some may argue that research during a global pandemic should only be conducted with either previously acquired or secondary data; others may argue that primary data collected voluntarily from willing participants is entirely ethical. Either way, respecting participants and their role in our research is always necessary. Beyond the requirements of doing so to uphold institutional research integrity expectations, it is our individual responsibility to ensure we, as researchers, are protecting those who make our work possible by assessing vulnerability, minimizing risk, and enhancing benefit, of participation – to the full extent of our capabilities.

covid-19, featured

Researching together: the value in collaboration when researching poverty during COVID-19

By Kayleigh Garthwaite, Ruth Patrick, Maddy Power, and Geoff Page

At the beginning of March, as universities started to close their campuses and it emerged we wouldn’t be returning to our offices for some time, it became clear that our usual practices and ways of carrying out research were about to change massively. Some of us will have been thinking about how to shift our existing research online, or wondering how to best document the effects of the pandemic. Others were considering if they should even be doing research at all.

Rapid response calls for research funding were everywhere. We started working together on a bid for the Nuffield Foundation’s call, which focused on social scientists conducting COVID-19 research in real-time to capture people’s experiences of the social, cultural, and economic impacts of the pandemic. When bringing the bid together, we were conscious of the many ethical issues connected with carrying out a research project on poverty with a participatory focus during a pandemic. While we were nervous – and at times uncertain – about the value of what we were planning, we were also acutely aware of the significant pressures families on a low income were facing due to the pandemic. We wanted to use what skills we had to try and ensure that their needs, experiences, and the appropriateness or otherwise of the policy response was placed firmly on the political agenda. We knew other researchers would also be cautious of how to navigate this in their own work on poverty and social security, too.

We started working on the Covid Realities project in April 2020. The project is exploring how families in poverty with dependent children are experiencing the pandemic, while also tracking how the social security system responds. We have sought to create a safe, online space for parents and carers to document their own experiences, and are also working with Child Poverty Action Group to draw on evidence emerging from their contact with front line welfare rights advisers, through their Early Warning System.

A significant strand of the project is working collectively with other researchers on the ‘COVID-19 and low-income families: researching together’ element of the project. This focuses on working closely with a range of research teams already undertaking fieldwork across the UK with families in poverty to support the generation of data specifically on COVID-19, and the synthesising and dissemination of relevant findings to policy makers and other key audiences. This is no easy task; working across 15 different research projects, and various research teams both inside and outside of academia requires a serious amount of planning, consideration, and above all, time. There is a huge value of working together and emphasising key findings across our diverse set of projects. However, important concerns can arise over data ownership, outputs, and key messaging, which need to be carefully thought through on an ongoing basis. But this process is already proving to be a really important way of collaborating, at a time when we are adhering to social distancing measures and working remotely.

We’ve already worked together to submit evidence submissions to parliamentary inquiries on the impact of COVID-19, drawing on emerging findings across our diverse studies. Conducting ethical research into poverty at this time means we need to try to create clear and effective chains of policy making engagement and dissemination. We know this isn’t easy; but we’re all doing our best to make sure that evidence generated can help inform current and future policymaking. Making sure we also include the voices and experiences of families in this process is central to fulfilling the aims of our project in communicating research evidence effectively and ethically.

Creating a space for researchers to think through some of these issues was something we wanted our project to be able to provide. Through this project, we want to offer researchers the support, tools, and resources to collectively think through how, and indeed whether, to carry out research on poverty during the pandemic. The idea is to create a place for honest discussions about what has or hasn’t worked well; a forum in which we can consider ethical debates and dilemmas together.

As part of this, we’re hosting bi-monthly webinars exploring topics that are central to researching poverty during COVID-19, and we’re hosting an ongoing blog series to unpack key issues that researchers are grappling with. So far, we’ve hosted blogs on whether we should be doing research during the pandemic at all, alongside others that have reflected on how we can be sensitive to participant fatigue around COVID-19 – should we really be asking people more questions about it when many people will associate the virus with fear, trauma and grief, or might just be fed up of thinking about it at all?

Already, it’s obvious there is a genuine interest and real value in trying to think through these ethical, practical, and methodological challenges together. There is a very real danger that requests to take part in research could be experienced as insensitive and inappropriate given the scale of the demands and pressures people are facing on a daily basis. We’ve written elsewhere about our concerns over placing additional pressures or strains on low-income families at a time of uncertainty, when they may be experiencing both physical and mental ill health, worsened hardship, and could possibly be grieving for lives lost in the pandemic.

We also need to fully consider the emotional impact on researchers themselves. Maintaining the role of researcher requires more emotional effort than before. What do we do when the interview is over? Our support mechanisms of colleagues and friends in the office next door are no longer there. Since beginning the project, we’ve had emails from people saying they’ve appreciated the chance to connect through our webinars, and to think through some of the issues they’re facing by writing a blog. Working at home can be isolating, stressful, and uncertain. Hopefully this part of our project will continue to be a collective space that helps the research community to think through, together, how we can best carry out ethically responsible research during COVID-19 – and indeed if we need to or should be doing this at all.

We would really like to hear from researchers on any of the ethical, practical, methodological – and emotional – challenges of researching poverty in the pandemic, so please do get in touch with us if you’d be interested in writing a blog or being part of our ongoing conversation.

COVID Realities is also working with parents and carers living on a low income to document and share their experiences. Find out more at www.covidrealities.org

The project has been funded by the Nuffield Foundation, but the views expressed are those of the authors and not necessarily the Foundation.