featured, Notebook

I Say, They Say: Effects of Providing Examples in a Survey Question

By Eva Aizpurua, Ki H. Park, E. O. Heiden & Mary E. Losch

One of the first things that survey researchers learn is that questionnaire design decisions are anything but trivial. The order of the questions, the number of response options, and the labels used to describe them can all influence survey responses. In this Research Note, we turn our attention to the use of examples, a common component of survey questions. Examples are intended to help respondents, providing them with information about the type of answers expected and reminding them of responses that might otherwise go unnoticed. For instance, the 2020 U.S. National Health Interview Survey asked about the use of over-the-counter medication, and included “aspirin, Tylenol, Advil, or Aleve” in the question stem. There are many other examples in both national and international surveys. Despite the potential benefits of using examples, there is a risk that respondents will focus too much on them, at the expense of overlooking cases not listed as examples. This phenomenon, called the “focusing hypothesis”, is what we test in our study.

Using an experimental design, we examined the effects of providing examples in a question about multitasking (“During the time we have been on the phone, in what other activities, if any, were you engaged [random group statement here]?”). In this experiment, respondents were randomly assigned to one of three conditions: the first group received one set of examples (watching TV or watching kids), the second group received a different set of examples (walking or talking with someone else), while the final group received no examples. Our goal was to determine whether respondents were more likely to report an activity (e.g., watching TV or walking) when it was listed as an example. We also wanted to understand whether providing examples resulted in respondents listing more activities beyond the examples.

We embedded this experiment in a telephone survey conducted in a Midwestern U.S. state and found support for the focusing hypothesis. As anticipated, respondents were more likely to mention the activity if it was provided to them as an example. However, the effect sizes were generally small and examples did not have an effect on the percentage of respondents who identified themselves as multitaskers, nor on the number of activities reported by them. This is because people faced with the experimental conditions were more likely to list the examples presented to them (i.e., watching TV, watching kids, walking, talking with someone else), while those in the control group more frequently reported activities outside this range (cooking, doing housework…), yielding no differences on the frequency of multitasking or on the number of multitasking activities.  Although examples can help respondents understand the scope of the question and remind them of certain responses, the results from this study indicate that they can also restrict the memory search to the examples provided. This has implications for survey practice, suggesting that the inclusion of examples in questions should be carefully considered and limited to certain situations, such as questions in which recall errors are anticipated or when the scope of the question might be unclear.

To learn more, see full IJSRM article here.

covid-19, Notebook

Adapting research with men during COVID-19: Experiences shifting to mobile phone-based methods

By Joe Strong, Samuel Nii Lante Lamptey, Richard Nii Kwartei Owoo, and Nii Kwartelai Quartey

It is impossible to understand masculinities without social research methods. Speaking and interacting with men is the fundamental cornerstone of the project Exploring the relationships between men, masculinities and post-coital pregnancy avoidance. Conducting these methods through ‘non-social’, distanced means, as a response to COVID-19, presents new challenges and opportunities and ethical considerations.

The original research sample frame was men aged 16 and over, who slept (proxy for ‘resident’) for at least some of their time in the study area. The research team were predominantly based / resident in the study area [a suburb of Accra], and all were living in Ghana prior to the declaration of a pandemic in 11 March 2020.

Response to COVID-19

The original research design necessitated close contact between respondents and the research team, using a household survey, focus group discussions and in-depth interviews. This proximity was quickly deemed unacceptable when compared to public health best practice (social distancing, limited movement, etc). Such methods endanger the respondents and the research team.

As it became evident that the pandemic was long-term, the team discussed potential mechanisms through which to continue the research in a safe and responsible manner. Mobile phone technology emerged as the only feasible way to ensure that social distancing and limited movement would be required for the research project to continue.

In the study area, mobile phone use is relatively high, reflecting broader trends in Ghana. However, these mobile phones were not all ‘smart’, i.e., it could not be assumed that respondents would have access to data or internet on their mobile devices. As such, continuing person-to-person survey interviews was the most feasible way, so as not to limit the sample to a) access to smart technology and b) ability / desire to navigate an online survey.

Thus, focus group discussions were removed entirely from the research design, as these could not be facilitated meaningfully through non-smart mobile phones. The survey questionnaires and in-depth interview schedule could remain the same, with additional questions on the impact of COVID-19. These had been tested prior to the pandemic in person to check for consistency, comprehension and relevance.

Practicalities

Obtaining equipment for the team in a timely and safe manner was essential – this included a mobile phone and three sim cards for each of the major telecommunication networks in the area. Fortunately, the team each had smart phone technology that allowed for communications to continue over WhatsApp.

Ethical amendments were submitted to account for consent being provided verbally, as written consent required inappropriate close contact. A huge outcome of the ethical amendment was the removal of anyone who could not consent for themselves. This has serious implications for the inclusivity of and representativeness of this research. The nature of the gatekeeping could not be observed or accounted for over the mobile phone. For example, it would not be clear if the parent of an adolescent – who required parental consent – would be in the room listening in. Critical voices, such as adolescents, people who need assistance with communication, e.g. sign language interpreters, are also not able to be incorporated into the survey.

The household listing conducted prior to the pandemic did not collect mobile phone information, as retrieving mobile numbers for each household member would be cumbersome and invasive. Thus, no sampling frame was available for the survey. To mitigate this, the study uses respondent driven sampling, whereby each survey respondent is asked to recruit three people from their personal network to be surveyed next and is compensated per successful recruit as well as for their own survey.

The experience of new methods

The use of mobile phones allows the respondents to decide when and where they want to be surveyed, providing them with greater autonomy than a household survey. In many ways, it empowers the respondent to have much more control over the survey. However, this also can make it harder, as the lack of physical presence makes distraction / missing a call much easier.

Moreover, the element of “out of sight, out of mind” hinders the efficiency with which respondents might recruit their friends, and the additional effort of conducting this recruitment through mobile phones might not help. We created regulations – no calling the same person twice in one day if they picked up, no more than three times in one week, end contact if asked – to try and mitigate overburdening respondents with reminders that might feel harassing.

We are finding that some respondents are reticent to be interviewed over the phone, preferring face-to-face interviews so that they might see the interviewer and build trust through sight. Despite the easing of lockdown in Ghana on 20 April 2020, the decision was made to maintain strict protocols of distancing between data collectors and respondents. This reflects the causes behind the ease of lockdown and that our research is non-essential, and we have a duty to avoid risking ourselves and the respondents. 

Responses to the lack of face-to-face cues were mixed. It makes it harder to use e.g. body language to gauge the respondent experience of the survey. On the other hand, it preserves a greater sense of anonymity for the respondent. It is necessary that data collectors “check-in” on respondents during an interview to ensure that the interview questions are not causing undue harm or stress, and that respondents be reminded that they are in control of the interviews. It is important that we acknowledge that the mobile phone becomes a part of the “context” of the research and it is essential to reflect on the impact of this.

Such experiences provide important opportunities for learning. Generally, we are finding that men are not afraid to talk to us over the phone. But we must acknowledge how many more men will be excluded through these methods and consider opportunities for their future inclusion. The greater control respondents have in arranging interviews to suit themselves is an important reminder of the need for patience and respect for respondents’ priorities and the (non-)essentialness of research.

At the time of writing (30 July 2020), 73 respondents have completed interviews, not including 22 seeds. For ongoing data visualisations and sneak peaks, visit the project website at: https://www.masculinitiesproject.org/

covid-19, featured, Notebook

Are novel research projects ethical during a global pandemic?

By Emily-Marie Pacheco and Mustafa Zaimağaoğlu

The global pandemic has inspired a plethora of new research projects in the social sciences; scholars are eager to identify and document the many challenges the COVID-19 situation has introduced into our daily lives, and explore the ways in which our societies have been able to thrive during these ‘unprecedented times’. Given the wide acknowledgement that life during a global pandemic is often more difficult than in our pre-pandemic circumstances, researchers must consider whether asking those in our communities to donate their time and energy to participating in our research is acceptable. Does recruitment for research which seeks to explore the psychological wellbeing and adjustment of those living through uniquely challenging circumstances during COVID-19 really reflect research integrity?

There is no simple answer to whether asking people to share their stories and experiences of COVID-19 is ethical or improper. Many would argue that social research has the potential to contribute many vital insights about life during a global pandemic which are unique to the humanistic lens and approach often reserved for the social sciences; such investigations could propel scholarly dialogue and manifest practically in recommendations for building resilient societies. However, social scientists have a responsibility to protect their participants from any undue harm they may experience as a result of their participation in a study. Thus, while social research may be especially important during a global pandemic, traditional study designs need to adapt to the circumstances of the pandemic and be held to higher ethical expectations by governing bodies and institutions.

Ethical social research during a global pandemic is reflected in research methods which demonstrate an awareness that we are asking more of our participants than ever before. Simple adaptations to existing projects can go a long way in bettering the experience of participants, such as by providing prospective participants additional information on what is expected of them if they choose to participate in a study – whether it be an online survey or an interview. Projects which aim to collect data using qualitative or interpersonal methods should be especially open to adaptation. These studies may be more ethically conducted by offering socially distant options, such as online focus groups or telephone interviews; adopting multimethod approaches and allowing participants the opportunity to contribute to projects in a medium which is most suitable for them may also be an ideal approach, such as by allowing participants the option to participate in online interviews or submitting audio-diaries conducted at their own discretion.

Attention should also be given to the various details of the research design which pertain to participant involvement more specifically. Does that online survey really needto include fifteen scales, and do they really need to ask all thosedemographic questions? Do online interviews really need to exceed thirty minutes and is it really necessary to require participants to turn their cameras on (essentially inviting you into their homes)? The ‘standard procedures’ for collecting data should be critically re-evaluated by researchers in consideration of the real-world context of those from whom they wish to collect data, with the aim of upholding their commitment to responsible research practices. Ethics boards should also aid researchers in identifying areas of their research designs which may be adapted to protect participants. This additional critical perspective may highlight participation conditions that may be arduous for participants, but which may have been overlooked as part of a traditional research design. 

Research during unprecedented times should also aim to provide a benefit to participants who generously donate their time and energy despite experiencing various transitions and changes in their own personal lives. While some researchers may need to devise creative solutions to meet this aim, many research methods in the social sciences have the inherent potential to serve as an activity which provides a benefit to those who engage in their process. For example, researchers may opt to collect data through methods which have a documented potential for promoting psychological wellbeing, or which are also considered therapeutic mechanism. Such approaches include methods which ask participants to reflect on their own experiences (e.g., audio-diaries, reflective entries, interviews with photo-elicitation) and those which focus on positive thoughts or emotions (e.g., topics related to hope, resilience, progress). Beyond these recommendations, researchers should also consider whether they really need participants at all. There are many options for conducting valuable research with minimal or no contact with participants, such as observational methods, content analyses, meta analyses, or secondary analyses. Some may argue that research during a global pandemic should only be conducted with either previously acquired or secondary data; others may argue that primary data collected voluntarily from willing participants is entirely ethical. Either way, respecting participants and their role in our research is always necessary. Beyond the requirements of doing so to uphold institutional research integrity expectations, it is our individual responsibility to ensure we, as researchers, are protecting those who make our work possible by assessing vulnerability, minimizing risk, and enhancing benefit, of participation – to the full extent of our capabilities.

Notebook

Online qualitative surveys?!?

By Virginia Braun and Victoria Clarke

A qualitative survey? What about face-to-face interaction? All the non-verbal cues? Probing and following up? Depth of data? These are the types of sceptical questions we hear a lot when we talk about our research using qualitative surveys. Our doctoral students have even been told at that they must supplement their qualitative survey data with another data source such as interviews, otherwise they will not have the depth of data they need. Sceptical questions like these are partly what motivated us to write about our experience of using online qualitative surveys for the International Journal of Social Research Methodology (LINK TO PAPER). We were also motivated by our enthusiasm for this method and wanted to share with other social researchers why we think it is a valuable addition to their methodological toolkit. We’ve used qualitative survey data over the last decade or so to explore everything from students’ responses to a gay pride T-shirt (Clarke, 2016, 2019) to male body hair removal discourse (Terry & Braun, 2016). We have also supervised numerous students using surveys – including Elicia Boulton, Louise Davey and Charlotte McEvoy, our three co-authors on this paper.

Examples of exclusively, or predominantly, qualitative surveys are relatively rare, but predominantly quantitative surveys with a few ‘open-ended’ questions are common. So how did we come to develop an enthusiasm for surveys as a qualitative method? Here we must credit our inspirational PhD supervisors – Celia Kitzinger and Sue Wilkinson – both great methodological innovators and ‘early adopters’, who encourage their PhD students to ‘experiment’ with research methods. Indeed, the small body of empirical research based on qualitative survey data mostly comes from Celia and Sue’s PhD students (e.g. Frith & Gleeson, 2004; Peel, 2010; Toerien & Wilkinson, 2004), and their students in turn (e.g. Hayfield, 2013; Jowett & Peel, 2009; Terry & Braun, 2017).

What is a qualitative survey then? Usually a series of questions focused around the topic of interest that participants answer in their own words. But qualitative surveys are not limited to questions and written responses, other possibilities include drawing tasks (see Braun, Tricklebank & Clarke, 2013) and responding to stimulus materials such as audio and video clips. Qualitative surveys are necessarily self-administered – if they were administered by researcher, they would essentially be a rather structured qualitative interview that would fail to reap the benefits of ‘messy’, participant centred qualitative interviewing. Qualitative surveys can be delivered in a variety of formats (hardcopy by post or in person, email attachment) but delivery via online survey software is pretty much the norm now, and that delivery mode is the focus of our discussion in our IJSRM paper.

When we think of (quantitative) surveys – as the sceptical questions we opened with illustrate – we typically think of breadth and more prosaically, larger samples. Whereas qualitative research is typically associated with depth and small, situated samples. How then can a method typically associated with breadth, and quantitative research, have anything to offer qualitative researchers? To appreciate the possibilities of qualitative surveys, we first need to recalibrate how we think of depth – shifting from associating it with individual data items, as is typically the case, to assessing depth and richness in terms of the dataset as a whole. This is not to say that individual survey responses can’t be rich, they can, and we include a powerful example in our paper from Elicia Bolton’s survey of experiences of sex and sexuality for women with obsessive compulsive disorder. Not all responses will be like this though – well, certainly not in our experience of using qualitative surveys so far. But an entire dataset of 60, 80 or a hundred responses will provide a rich resource for qualitative analysis. Survey data also have their own unique character, they are not simply like reduced interview data. They are very focused, dense with information – to the extent that a dataset that runs to the same number of pages as a small number of interview transcripts can feel like a lot of data! Our students typically cycle through an initial panic at the start of data collection or piloting – the responses aren’t very detailed! – to feeling delighted, or even overwhelmed, by the amount of information in the final dataset.

Okay, so survey data can be rich, but why would I use a qualitative survey though, rather than do some interviews over Zoom or Skype, with all the advantages of virtual interviewing? Let’s start with some of the practical and pragmatic benefits of qualitative surveys – for us as researchers. There are no bleary-eyed video calls at 6am or 11pm. Data collection can be relatively quick – and there’s no transcription! – leaving plenty of time for data analysis, which is particularly useful if working to a tight or fixed deadline. We note that we are not advocating for quick (and dirty) as inherently good, however; good quality qualitative research takes time, and using a qualitative survey can allow time for the slow wheel of interpretation to turn when we do not have all the time we would ideally want and need to complete our research. In research with no funding, there are few or no costs associated with data collection (especially if you have access via your institution to online survey software). When it comes to student research, we think qualitative surveys can open up research possibilities – because there is no direct interaction with participants, there are likely fewer ethical concerns around inexperienced researchers addressing sensitive topics. For example, one of our undergraduate students researched young adults’ experiences of orgasm using a qualitative survey – it’s highly unlikely they would have received ethical approval to research this using interviews (see Opperman, Braun, Clarke & Rogers, 2013).

For participants, there are even more practical benefits – not least that they can participate when it is most convenient for them. Louise Davey noted that her participants often completed her survey on experiences of living with alopecia early in the morning or late in the evening; unlikely times for an interview. Online survey software will also usually allow completion over multiple sessions, so participants can complete the survey in several short bursts, fitting participation around their schedule, commitments, and indeed energy. This is one of the ways in which online qualitative surveys can give participants a greater sense of control over their participation. Surveys also typically ask less of participants – they do not have to spend an hour or two talking to a researcher at a particular time, they do not have to travel to meet a researcher in person. They also have the advantage of a strong sense of felt-anonymity (in practice, online qualitative surveys are not completely anonymous) – this can be vital for some topics. In Charlotte McEvoy’s research, for instance, on therapists’ views on class and therapy, some participants commented that they were glad of the anonymity of the survey, they would not have shared what they did – and we can speculate, perhaps even not participated at all – if they were invited to take part in an interview. This connects to another advantage of qualitative surveys – that they have the potential to open up participation for groups for whom face to face participation is challenging in various ways. This includes some disabled people, people with caring responsibilities, people with visible differences – such as alopecia – who may feel anxious about being visible to and open to scrutiny by the researcher, and people for whom social interaction with strangers can be profoundly anxiety inducing (such as people with OCD).

This is just a taster of some of the benefits and possibilities of qualitative surveys. We hope we have enticed you to read further about qualitative survey literature and discover the joys, and challenges, of this method for yourself!

See full IJSRM article here.

covid-19, featured

Researching together: the value in collaboration when researching poverty during COVID-19

By Kayleigh Garthwaite, Ruth Patrick, Maddy Power, and Geoff Page

At the beginning of March, as universities started to close their campuses and it emerged we wouldn’t be returning to our offices for some time, it became clear that our usual practices and ways of carrying out research were about to change massively. Some of us will have been thinking about how to shift our existing research online, or wondering how to best document the effects of the pandemic. Others were considering if they should even be doing research at all.

Rapid response calls for research funding were everywhere. We started working together on a bid for the Nuffield Foundation’s call, which focused on social scientists conducting COVID-19 research in real-time to capture people’s experiences of the social, cultural, and economic impacts of the pandemic. When bringing the bid together, we were conscious of the many ethical issues connected with carrying out a research project on poverty with a participatory focus during a pandemic. While we were nervous – and at times uncertain – about the value of what we were planning, we were also acutely aware of the significant pressures families on a low income were facing due to the pandemic. We wanted to use what skills we had to try and ensure that their needs, experiences, and the appropriateness or otherwise of the policy response was placed firmly on the political agenda. We knew other researchers would also be cautious of how to navigate this in their own work on poverty and social security, too.

We started working on the Covid Realities project in April 2020. The project is exploring how families in poverty with dependent children are experiencing the pandemic, while also tracking how the social security system responds. We have sought to create a safe, online space for parents and carers to document their own experiences, and are also working with Child Poverty Action Group to draw on evidence emerging from their contact with front line welfare rights advisers, through their Early Warning System.

A significant strand of the project is working collectively with other researchers on the ‘COVID-19 and low-income families: researching together’ element of the project. This focuses on working closely with a range of research teams already undertaking fieldwork across the UK with families in poverty to support the generation of data specifically on COVID-19, and the synthesising and dissemination of relevant findings to policy makers and other key audiences. This is no easy task; working across 15 different research projects, and various research teams both inside and outside of academia requires a serious amount of planning, consideration, and above all, time. There is a huge value of working together and emphasising key findings across our diverse set of projects. However, important concerns can arise over data ownership, outputs, and key messaging, which need to be carefully thought through on an ongoing basis. But this process is already proving to be a really important way of collaborating, at a time when we are adhering to social distancing measures and working remotely.

We’ve already worked together to submit evidence submissions to parliamentary inquiries on the impact of COVID-19, drawing on emerging findings across our diverse studies. Conducting ethical research into poverty at this time means we need to try to create clear and effective chains of policy making engagement and dissemination. We know this isn’t easy; but we’re all doing our best to make sure that evidence generated can help inform current and future policymaking. Making sure we also include the voices and experiences of families in this process is central to fulfilling the aims of our project in communicating research evidence effectively and ethically.

Creating a space for researchers to think through some of these issues was something we wanted our project to be able to provide. Through this project, we want to offer researchers the support, tools, and resources to collectively think through how, and indeed whether, to carry out research on poverty during the pandemic. The idea is to create a place for honest discussions about what has or hasn’t worked well; a forum in which we can consider ethical debates and dilemmas together.

As part of this, we’re hosting bi-monthly webinars exploring topics that are central to researching poverty during COVID-19, and we’re hosting an ongoing blog series to unpack key issues that researchers are grappling with. So far, we’ve hosted blogs on whether we should be doing research during the pandemic at all, alongside others that have reflected on how we can be sensitive to participant fatigue around COVID-19 – should we really be asking people more questions about it when many people will associate the virus with fear, trauma and grief, or might just be fed up of thinking about it at all?

Already, it’s obvious there is a genuine interest and real value in trying to think through these ethical, practical, and methodological challenges together. There is a very real danger that requests to take part in research could be experienced as insensitive and inappropriate given the scale of the demands and pressures people are facing on a daily basis. We’ve written elsewhere about our concerns over placing additional pressures or strains on low-income families at a time of uncertainty, when they may be experiencing both physical and mental ill health, worsened hardship, and could possibly be grieving for lives lost in the pandemic.

We also need to fully consider the emotional impact on researchers themselves. Maintaining the role of researcher requires more emotional effort than before. What do we do when the interview is over? Our support mechanisms of colleagues and friends in the office next door are no longer there. Since beginning the project, we’ve had emails from people saying they’ve appreciated the chance to connect through our webinars, and to think through some of the issues they’re facing by writing a blog. Working at home can be isolating, stressful, and uncertain. Hopefully this part of our project will continue to be a collective space that helps the research community to think through, together, how we can best carry out ethically responsible research during COVID-19 – and indeed if we need to or should be doing this at all.

We would really like to hear from researchers on any of the ethical, practical, methodological – and emotional – challenges of researching poverty in the pandemic, so please do get in touch with us if you’d be interested in writing a blog or being part of our ongoing conversation.

COVID Realities is also working with parents and carers living on a low income to document and share their experiences. Find out more at www.covidrealities.org

The project has been funded by the Nuffield Foundation, but the views expressed are those of the authors and not necessarily the Foundation.