Calls, featured, Notebook

Remote qualitative data collection: Lessons from a multi-country qualitative evaluation

By Mehjabeen Jagmag *

Like most researchers who had planned to begin their research projects earlier this year, our research team found our data collection plans upended by the pandemic. We had designed our research guides, received ethical clearance and completed training our research teams for a multi-country endline evaluation of an education programme in Ghana, Kenya and Nigeria much before we heard of the COVID-19 pandemic.

A few days before our teams travelled to their respective data collection sites, phone calls started pouring in – schools were closed indefinitely, travel between cities was restricted, and we were beginning to understand how much the COVID-19 pandemic would change our lives. After a few weeks of waiting and watching, it became apparent that we could not continue in-person data collection.

We revised our research guides and prepared ourselves for conducting remote phone-interviews with our research participants. Given that this was the third and last round of data collection in our multi-year panel research, we had previously collected phone numbers of our research participants and acquired permission to be able to contact them on the phone for further research. We set up remote research desks for the team and began preparation for data collection.

What we were unsure about was whether our research plans would be successful. Accounts of fraudulent callers promising medical remedies and peddling fake health insurance packages had made people wary of responding to unknown phone numbers. We were not sure how many of the phone numbers we had collected in the previous year would still be working, and most importantly, we were not sure how our research participants were faring under the lockdown and whether they would want to speak with us. Finally, our research participants included primary school students, who were an essential voice in our study. We were keen to conduct interviews but were not sure if this would be feasible – would parents trust us enough to speak to us and consent to their children speaking to us? Once we secured consent from parents, would children provide assent? As trust was the key element to completing our research successfully, we devised a data collection plan that included the following elements, that are likely to be necessary for future remote data collection.

Training and retraining for remote data collection

We spent time discussing as a team what the potential challenges may be and how we plan to respond to them. We drew up a collective list of answers that we could draw on to communicate clearly and effectively about the evaluation, respond to any queries and alleviate any concerns that our participants had. This list and knowledge grew, and we collected data, and debrief meetings with the teams at the end of each data helped ensure this was a live document.

Seek feedback from key informants

We contacted community leaders and headteachers to enquire about how we should approach data collection with school and community participants. They provided important contextual information that was occasionally specific to each community. We used this information to improve our introductory messages, the time and dates we called and how we approached research participants.

Seek introductions from trusted leaders

We also asked community leaders and headteachers to support our recruitment process by sending messages to the participants about our research before it began. Doing so helped minimise any uncertainty of the veracity of our calls. Where relevant, we compensated them for their airtime.

Give participants time to prepare for the interview

We shared information about our organisation and the research objective over text messages or calls, which gave research participants enough time to decide whether they wanted to participate. It also helped plan to speak at a time would suit them best for a discussion, and also consult with their family and children if they wanted to participate in the research.

Ensure continuity of research teams

As this was an endline evaluation, we had research team members who participated in previous rounds of data collection calling the participants they were likely to have visited in the past. Where this was possible, it increased trust and facilitated easy conversations.

Prepare case-history notes

We prepared short case history notes about the programme and school and what we had learned from previous research rounds for each school to build confidence that our intentions and credentials were genuine. These notes helped remind research participants of our last conversation, helped us focus on what has changed since that last conversation, which in turn helped keep conversations short and in general proved to be a useful conversation starter.

Save time at the beginning and end for questions

We ensure research participants had enough time to ask us about the programme, our motivations, go over the consent form, understand why we wanted to speak with the children or for children to ask parents for their permission before we began our interviews. To ensure that that the conversation did not feel rushed, we designed shorter research guides.

Plan for breaks or changes when interviewing with young participants

When speaking with students, we anticipated time to break and distractions during the call, which helped maintain a relaxed pace during the interview. If students were uncomfortable with phone interviews, we, eased the conversation to a close to minimise any distress caused to the participant.

Summary and Conclusion

We completed data collection in all three countries, albeit with a less ambitious research plan that we originally intended for an in-person research study. The key objective of our research was to collect the optimal amount of data that would inform the programme evaluation while making the interview process convenient and comfortable for the research participants involved. To do so, we have learned that it is vital for participants to have confidence in the researchers and the motive for collecting data. Planning before we began data collection and updating our knowledge as the research progressed proved invaluable to our experience.

* Mehjabeen Jagmag is a Senior Consultant with Oxford Policy Management.

covid-19, featured, Notebook

Critical reflections on the ‘new normal’: Synchronous teaching of CAQDAS-packages online during COVID-19

By Christina Silver, Sarah L. Bulloch, & Michelle Salmona

Our contribution discusses synchronous online teaching of digital tools for qualitative and mixed-methods analysis, known as Computer Assisted Qualitative Data AnalysiS (CAQDAS) packages, during the COVID-19 pandemic. Teachers must take responsibility for, and be sensitive to, the current additional challenges and pressures upon learners and attend to them effectively. Learners are never homogenous but, in these contexts, their heterogeneity and personal situations bring our responsibilities as teachers into sharper focus.

Challenges of teaching CAQDAS-packages

Teaching CAQDAS-packages is challenging as research methods and technology are taught together, and researchers often need support overcoming hurdles associated with integrating technology into research practice. Although it can support critical reflection on methods-driven research, novice researchers have trouble connecting method and software (Salmona & Kaczynski, 2016; Schmieder, 2020).

Traditionally CAQDAS is taught in-person but even before COVID-19, there was a gradual move to online courses, which can be cost-effective and reach wider groups. However, teaching CAQDAS online has its own challenges, including possible technical problems, catering to different learning styles, and interactional issues (Kalpokaite & Radivojevic, 2020). Learning CAQDAS-packages online also heightens challenges in overcoming barriers to successful technological adoption due to the lack of support normally present in-person (Salmona & Kaczynski, 2016). Teaching CAQDAS-packages online during COVID-19 poses additional challenges related to learner availability, real-life bleeding into the classroom, and resultant interactional issues. 

Learner availability in the COVID-context 

Pre-COVID-19, both in-person and online, certain assumptions were often made concerning the ‘availability’ of learners: 

  • They would be present for the duration, unless specific exceptions were brokered; e.g. warning they may have to take a call or leave early.
  • Only registered learners would be present – not family-members, carers, or dependents as well. 
  • Learners would be in a state of mental and physical health suited to learning.

Teachers could generally assume to be engaging not with whole individuals, but with focused

“learners”: the mentally present and mentally well, physically present and physically well, the not-distracted, the captive from start to finish, solo individuals.

Real-life bleeding into the classroom

During COVID-19 these assumptions no longer hold true. We cannot expect learners to focus for the whole allotted time because they cannot necessarily physically or emotionally remove themselves from their home-life contexts. New distractions and stresses include: interruptions from household members, capacity to concentrate for lengthy periods of screen-time, and mental-health issues associated with being more isolated. However, because in-person interactions have largely vanished, learners are keen to participate in online sessions, despite the distractions and stresses. Online sessions also provide learning opportunities for those previously unable to access in-person events. 

As we teach and learn from our homes, real-lives bleed into the classroom. Sharing our images via video-stream allows others into our lives, which is potentially risky. We’ve found more learners choose not to share their video-stream than do, especially in larger groups and when they don’t know each other. 

What we miss by not ‘seeing’

Those used to teaching in-person can find this tricky, as the non-verbal cues used to judge learners’ progress are absent. CAQDAS teachers can no longer ‘walk-the-room’ observing learners’ computer-screens to identify those needing additional support. Screen-sharing can be a solution; but is more time-consuming and ethically difficult when working with confidential data, and impossible if using two devices (one to access the meeting, the other to operate the CAQDAS-package). We miss a lot by not seeing in these ways.  

One risk is that those who can actively participate inadvertently soak-up attention at the cost of those who cannot. It’s our responsibility as teachers to be aware of this and design creative solutions to enable every learner to participate as much as they are willing and able, whilst still benefiting from the session.

Adjusting tactics for the ‘new normal’

We’re therefore continually adjusting how we teach CAQDAS-packages online during COVID-19. Current uncertainties land responsibilities on us as teachers, not on our course participants: we must find out what they need, reflect on our practice, and refine our pedagogies. 

Moving from in-person to online always requires a redesign (Silver & Bulloch, 2020), but during COVID-19 we are also:

  • Educating ourselves about accessibility to ensure we sensitively and effectively open our events to every type of learner
  • Engaging learners more before sessions to understand personal/research needs and provide pre-course familiarisation materials
  • Reducing class-sizes. It’s often assumed class-sizes can be larger online, but we find the opposite, especially during COVID-19. Although we’ve recently experienced pressure to increase group size, we’re resistant because of the increased need to balance the requirements of every learner, and provide individual support 
  • By co-teaching we provide additional support in synchronous online events during COVID-19. Learners can be split according to their needs and two groups supported simultaneously
  • Providing more post-course resources to support learners’ continued use of CAQDAS-packages and hosting platforms for them to communicate with one another afterwards (e.g. VLE platforms)
  • Diversifying teaching tactics to provide as many opportunities as possible for learners to engage and participate. Awareness of different ways people learn has always been central to our pedagogies (Silver & Woolf 2015), but our sensitivities and reflections have increased. We’ve found mixing up tactics (see image) in shorter sessions more effective.

Where do we go from here?

Teachers continually critique and reflect on practice, but COVID-19 requires a re-evaluation of learners’ differences and reflection about their more challenging situations. We are all learning and must continue to do so.

COVID-19 brings ethical issues even more to the forefront, including the appropriateness of requiring or encouraging learners to share their image via video. We must think about disabilities, access to technology, and socio-economic issues in a context where learning is only available online. Positives have also emerged, as sessions can be followed from a range of devices and locations.

COVID-19 forces us to explicitly consider the well-being of learners. Despite coming at this difficult time, we welcome this focus. All our situations have changed, so we need to think about the issues differently. What are the additional ethical issues we must now address? How do we keep this conversation going?

About the authors

Together we have 50+ years experience teaching CAQDAS-packages and 30+ years experience teaching online. Dr Michelle Salmona is President of the Institute for Mixed Methods Research and an international consultant in: program evaluation; research design; and mixed-methods and qualitative data analysis using data applications. Michelle is also an Adjunct Professor at the University of Canberra, Australia specializing in qualitative and mixed methods research. Dr Sarah L Bulloch is a social researcher passionate about methods, with expertise in qualitative and quantitative analysis, as well as mixing the two. She has worked in academic, government, voluntary and private sectors. Sarah teaches introductory and advanced workshops in several CAQDAS packages as a Teaching Fellow for the CAQDAS Networking Project at the University of Surrey, as well as teaching quantitative analysis using SPSS. Dr Christina Silver is Director of Qualitative Data Analysis Services, providing training and consultancy for qualitative and mixed-methods analysis. She also manages the CAQDAS Networking Project (CNP), leading its capacity-building activities. She has trained thousands of researchers in the powerful use of CAQDAS-packages, including NVivo, and developed the Five-Level QDA® method with Nick Woolf.  

References

  • Kalpokaite, N. & Radivojevic, I. (2020). Teaching qualitative data analysis software online: a comparison of face-to-face and e-learning ATLAS.ti courses, International Journal of Research & Method in Education, 43(3), pp. 296-310, DOI:10.1080/1743727X.2019.1687666.
  • Salmona, M. & Kaczynski, D. (2016). Don’t Blame the Software: Using Qualitative Data Analysis Software Successfully in Doctoral Research. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research, 17(3), Art 11, http://nbn-resolving.de/urn:nbn:de:0114-fqs1603117.
  • Schmieder, C. (2020). Qualitative data analysis software as a tool for teaching analytic practice: Towards a theoretical framework for integrating QDAS into methods pedagogy. Qualitative Research, 20(5), pp. 684-702. 
  • Silver, C. & Woolf, N (2015) “From guided instruction to facilitation of learning: The development of Five-level QDA as a CAQDAS pedagogy that explicates the practices of expert users” International Journal of Social Research Methodology, Vol. 18, Issue 5. Pp527-543
  • Silver, C. & Bulloch, S.L (2020) Teaching NVivo using the Five-Level QDA(R) Method: Adaptations for Synchronous Online Learning. Paper presented at the QSR International Virtual Conference, Qualitative Research in a Changing World. September 24th 2020
covid-19, featured

Our lockdown walks: Physically, but not socially, distanced walking as method

By Lauren White and Katherine Davies

A shop window displaying the sign ‘See you soon’

In the UK, the first coronavirus national lockdown meant that many of us were permitted one form of exercise a day – a walk, a run, a bike ride – a pocket of freedom and an opportunity for outdoor life. In the Department of Sociological Studies at the University of Sheffield, this was the time that we usually plan for our annual sociological walk. Our walks seek to explore the role of everyday moments as a methodological route of inquiry, enabling us to be creative in academic discussions by taking them out of university rooms and thinking and learning with our feet (Ingold, 2004). In 2020, our discussions were now restricted to the virtual spheres of google and zoom, meaning our mobile methodologies required a rethink.

Determined to make the most of our mobile lives, we invited colleagues to share photographs and notes of their walks, runs and cycles throughout the lockdown. As a group we shared photographs of our families walking together, our dog walks, accounts of nature, sensory experiences of urban and rural places, and offered visual reflections of loss and future imaginaries. We joined together and discussed these online. Whilst this blog post is not about an active research project and its methodology per se, it offers a momentary reflection on walking during lockdown and the methodological possibilities we gathered from our ‘exercise’.

Through ‘taking a walk’ alone (Carpiano, 2009) but virtually discussing together, we found we were able to access many of the advantages associated with traditional go-along mobile methods. Through discussions with photographs, we were able to share the material experience of moving through place. For example, how we navigated obstacles, stepping to one side and keeping space as we experienced the world through our feet (Ingold, 2004).

Navigating obstacles as cars and bins take over the pavement

We were surprised by the ease with which we accessed socio-atmospherics (Mason, 2018), understanding one another’s sensory worlds and exploring together the uncanny feelings of the times. For example, we shared the unseasonable joy of a summer’s day and reflected on how this jarred with the knowledge that crisis was, quite literally, in the air.

A large tree with pink blossom


Though we were not together for our walks, we discussed their relational context and gained insights into personal lives and everyday mobilities. We learnt of our colleagues’ relationships with their communities, neighbours and their proximities to families and friends; from friendships formed between local dogs, to the rainbows displayed in neighbours’ windows. We were struck by the heightened role of doorstops, as we shared stories of the cakes and Easter eggs left on by the door in the absence of social contact. We shared our embeddedness and our emotional relationships with place. Talking about walking and lockdown walks was a proxy for personal discussions and a way into talk about our shared everyday sensitivities.

Hopscotch chalked on the pavement

Whilst these are brief reflections and not based on an active research project and carefully devised methodology, we propose that there are opportunities for capturing mobilities in everyday life, even in the absence of walking together. Echoing May and Lewis’ (2020) argument, we have demonstrated that it is not necessary to physically walk together to glean rich insights into embodied and sensory experiences of place. 

Walking and everyday journeys as a method is possible and offers promise for social research futures. Talking about mobilities and place virtually further opens up opportunities for those who cannot be co-present. Appreciating that walking and the ability to be outdoors is a privilege, not afforded to everyone (Rose, 2020), we invite researchers and practitioners to embrace the potential of virtual walking and go-along methodologies.

Walking alone but discussing together also offers a pedagogical tool for collective knowledge, with the potential to unlock the ‘mysteries’ of place (Bates and Taylor 2017: 20). The group nature of our discussion, along with the use of photographs, was crucial to the insights we gained and became a useful way into learning and knowing about relationships with place.

Such methodological reflections on physically distanced mobilities and virtual collective discussions can evoke the sociological imagination of mobile methods in physically distanc(ed) futures. Whilst our ability to be mobile side by side has been curtailed, we can learn through our feet as well as through our screens, together. And whilst our mobilities may have stopped at the doorstep or at the other side of a park bench, our virtual mobile accounts tell a story of relational lives in physically, but not socially, distanced times. 

A pond in the park with the shadows of trees

Acknowledgements: We would like to thank our colleagues and contributors from the Everyday Life and Critical Diversities research group for joining in on the discussions and sharing personal insights.

References

Bates, C. & Rhys-Taylor, A. (2017) Walking through social research. Abingdon: Routledge.

Carpiano, R. M. (2009) ‘Come take a walk with me: The ‘go-along’ interview as a novel method for studying the implications of place for health and wellbeing’, Health & Place, 15(1), pp.263-272.

Ingold, T. (2004) ‘Culture on the ground: The world perceived through the feet’, Journal of Material Culture, 9, 315-340.

Mason, J. (2018) Affinities: Potent Connections in Personal Life. Cambridge: Polity.

May, V. & Lewis, C. (2020) ‘Researching embodied relationships with place: Rehabilitating the sit-down interview’, Qualitative Research, 20(2), pp.127-142.

Rose, M. (2020) ‘Pedestrian practices: Walking from the mundane to the marvellous’, in H. Holmes & S. M. Hall. (Eds.) Mundane methods: Innovative ways to research the everyday. Manchester: Manchester University Press. Chapter 13, pp.211-229.

Calls, covid-19, featured, Notebook

Exploring unplanned data sites for observational research during the pandemic lockdown

By Areej Jamal (UCL Social Research Institute)

I was getting my PhD upgrade and preparing for my fieldwork soon after, when the Covid-19 outbreak struck. Due to the border closures, I was stuck in the country where I was undertaking my PhD and my fieldwork was due to start overseas which is also my home. I could not travel back home for months.

Not sure of when the borders would open again and anxious about starting my fieldwork, I had to adapt to the new circumstances and utilise my research time more efficiently. Since my research employs a mixed methods design using online surveys and qualitative interviews, I did not have to pivot much of the original design. However, I also had an observatory aspect in my proposed design. Being an insider to the community I am researching, I was looking forward to attending some social meetings to observe and gather the less apparent insights during my fieldwork time. But since the social distancing protocols of Covid-19 seemed long term, I had to reconsider the observation element. In this blog, I will reflect on how I took the changing circumstances in my stride and tried to gather insights from unplanned data collection sites.

Reframing the methods design

My research investigates lived experiences of long-term migrants and how they make sense of their identity and belonging in a country where they have no pathways to citizenship. Due to the Covid-19 crisis I had to reframe a few ways of collecting data ensuring to keep the essence and relevance of my research questions. Though I could carry out the online surveys as originally planned, I had to switch the in-depth interviews to online interviews. The ethical and methodological amendments arising out of online interviews were submitted to the ethics committee for review.

Being stranded abroad in lockdown, I started exploring and observing the online content as sites for data collection. The unobtrusive observation of online content became a very significant part of my revised research design. The two main online sources where I found salient insights were the online videos (vlogs) on YouTube where some migrants had been documenting almost every aspect of their lives and the other was online support groups where mostly distressed migrants were interacting. The latter was a chance discovery as I, myself was a part of these groups seeking guidance and information to return home.

Unobtrusive observation implies that a researcher observes and collects data from online sources such as websites; social media sites or discussion forums without necessarily interacting with the participants. I was passively observing the interactions taking place on these online platforms.

Making sense of the data and meanings emerging from the observation sites

Robinson (2016) and Seale et.al (2010) posit that unsolicited narratives provide richness of new knowledge that is often lacking in solicited accounts. Since the researched community is unaware of the ongoing observation, they tend to offer genuine and certain interpretations of life under specific circumstances. And so, the narrator controls the content without the researcher’s interference.

The migrant stories I had been observing from the selective YouTube accounts and the online support groups that I had joined on WhatsApp particularly offered me very significant source of information and insights. The self-reflections and opinions the observed groups expressed about their temporal migrant status and the impact it has had on some of their life decisions and which was further exacerbated by the pandemic gave me context and ideas relevant for my research questions. The narrator driven stories uncovered aspects of life, which probably would not have occurred in any of my other methods. Seale et.al (2010) suggests that since the online interactions occur in real time, they offer some sort of immediacy which is often lacking in methods where participants mostly reflect and reconstruct past occurrences.

The new knowledge emerging out of these online sources helped me draft some of the initial themes and areas to further investigate through the other methods.

Methodological and Ethical considerations

There are various ethical debates surrounding the unobtrusive research in existing literature and ways of examining personal narratives produced by individuals through online medium.  Some researchers (Eysenbach and Wyatt,2002, Seale et.al, 2010) argue that since most of the online content is publicly available aimed at general audience, the need for informed consent is ambiguous. However, in case of online support groups as Barker (2008) and O’Brien and Clark (2011) discuss the limitations of private content due to smaller number of groups members. The support groups I had been a member of, had admins and certain privacy protocols that all group members had to abide by.

The positionality of the researcher is a very important aspect throughout the research process. Salmons (2012) E-interviews Research Framework offers useful tips reflecting on some crucial questions of self-reflexivity when undertaking unobtrusive observations of the online content.

Matters of confidentiality and seeking consent for data generated from online resources needs much deliberation and largely depends on the objectives of the researcher and the ways they aim to present and report the findings. Although I am still exploring ways of representing information from these valuable sites of information, I find the idea of fabrication approach by Annette Markham (2012) quite useful, which implies ‘involving creative, bricolage-style transfiguration of original data into composite accounts or representational interactions’ (2012, p334) without divulging any specific details of the researched community.

Conclusions

Sometimes the most obvious data sites would not be as apparent unless faced with unexpected circumstances restraining our methodological choices. Agility is an intrinsic characteristic of most social research. At present, data collection is in constant flux responding to the unprecedented crisis. And now more than ever, the relentless pandemic situation offers a critical window for researchers to make every effort to explore creative and novel approaches of data collection and innovative ways to tap the potential of the existing methods.

Announcements, Calls, featured

Call for Contributions: Adapting Conventional Research Methods for the New Normal

In search of Novel Adaptations in Social Science Research Methods for the World of COVID

With a second wave of the global COVID-19 pandemic now sweeping across the West, it is becoming clear that returning to ‘normal‘ maybe a long way off yet. Though, the need for social research is now more pressing than ever. The International Journal of Social Research Methodology is inviting researchers, academics, and doctoral students to share blog contributions reflecting their experiences in adapting existing research methods to meet the needs of our new research paradigm. The goal is to help the social science research community find inspiration and learn from one other as we continue to adapt our methods to meet contemporary needs. Contributions reflecting all aspects of social science research, including research ethics as well as quantitative and qualitative methods, are welcome. Preference will be given to submissions that demonstrate novel adaptation, creativity, and/or innovation.

Your contributions should be emailed to tsrm-editor@tandf.co.uk in MS Word format. Accompanying images may be included. Contributions should not exceed 1,000 words.

Updated Deadline for submissions is 21 December 2020.