Announcements, featured

The winners of our ECR paper competition for 2020-21

We are pleased to announce the results of our 2020-21 IJSRM competition for papers written by early career researchers (ECRs) who, at the time of submission, were either doctoral students or in their first three years of post-doctoral employment. Our aim has been to encourage and recognise research and contributions from new scholars in current and emerging methodological debates and practice.

All entries were subject to the Journal’s usual refereeing processes and had to reach our normal publishing standard. The winners were selected by a sub-panel of members of the IJSRM Editorial Board and the Journal Editors. The panel were impressed with the very strong field of entries, and we are pleased to announce not only a winner of the ‘Best ECR Article’ but also three ‘highly commended’ runners up.

Our IJSRM Early Career Researcher Prize is awarded to Stefanie Döringer (Austrian Academy of Sciences and University of Vienna) for her article on ‘The problem-centred expert interview: Combining qualitative interviewing approaches for investigating implicit expert knowledge’. The panel of judges remarked on a ‘clearly written and illuminating account’, representing ‘a lightbulb moment that brings two previously disconnected traditions together’, and that will be ‘highly valuable as a reference for many researchers for years to come’. Stephanie’s article has already been viewed over 16,500 times.

Stephanie said, ‘It is a great honor for me that my paper is awarded with the IJSRM Early Career Researchers’ prize. The appreciative comments from the competition judges encourage me to follow my research interest further and to deepen my work with qualitative methods in social research’.

Our highly commended runners (in alphabetical order) are: Riccardo Ladini (University of Milan): ‘Assessing general attentiveness to online panel surveys: The use of instructional manipulation checks’ Órla Meadhbh Murray (Imperial College London): ‘Text, Process, Discourse: Doing Feminist Text Analysis in Institutional Ethnography’ Kate Summers (London School of Economic and Political Science): ‘For the greater good? Ethical reflections on interviewing the ‘rich’ and ‘poor’ in qualitative research

Many congratulations to Stephanie, and also to Kate, Órla and Riccardo.

Announcements

Changes to our Journal metrics – a more rounded picture?

Journal Impact Factors – based on citations of articles published in the journal concerned, have been used as a proxy for the prestige of a journal in comparison with others in its field. And promotion committee considerations can be based on whether an academic has had articles published in high impact factor outlets. But this means of assessing the value of journals, the quality of articles published in them, and by extension the standing of the authors of published pieces, has been subject to criticism. These concerns run from questioning the reliability of the measurement and ranking, through encouragement to editors and authors to game the system, to condemnation of a neo-liberalised audit culture in academia. Some publishers and platforms, such as PLOS, have decided not to display Impact Factors.

Our Journal’s publisher, Routledge/Taylor & Francis, is now starting to shift away from the Impact Factor as a key indicator of quality, replacing it with a ‘basket’ of metrics in an effort to provide a more rounded views of the various ways in which a journal and articles published within it may have scholarly, policy and social ‘impact’.

The metrics being posted on the publishers’ IJSRM page for 2020 are:

  • an Impact Factor of 3.061 for the year, and one of 4.508 over a 5-year period
  • a new Journal Citation Indicator of 2.15, ranking 13/254 in the category of Social Sciences, Interdisciplinary, and
  • a CiteScore of 5.0, ranking 16/260 in all Social

Whether or not such moves deal with the concerns about measurement reliability, gaming the system, and evaluating of academics, is a moot point.

featured, Notebook

Using objects to help facilitate qualitative interviews

by Signe Ravn

Doing empirical research on imagined futures is a methodological challenge. As scholars have argued, generating rich insights into how such futures might look can be difficult as participants may produce somewhat generic or stereotypical accounts of what the future might hold or even refuse to engage in such tasks (which of course provides other insights). Over the past decade, these challenges have led many qualitative researchers to explore different forms of creative, arts-based and/or participatory methods to approach the topic in new ways. In some cases, these approaches have been productive, and in other cases they lead to new questions about how to then interpret the findings. And sometimes they don’t really generate more concrete insights after all.

In my longitudinal research on the everyday lives and imagined futures of young women with interrupted formal schooling, I also used various creative methods to break away from the traditional interview format and to seek to approach the ways in which participants imagined their futures from multiple different perspectives. This approach was inspired by Jennifer Mason’s work on facet methodology. In my recent paper for the International Journal of Social Research Methodology I explore one creative method that proved particularly fruitful, that is, an object-based method. In brief, this method was deployed in the third interview with my participants (after one year) and involved asking participants to bring ‘one thing (like a gift, some clothing, a thing you once bought, or something else) that reminds you of your past and a thing that you relate to your future’. Only one participant asked for a clarification of what these items could be, while the remainder were happy to do this task, and some even said right away that they knew exactly what to bring. On the day of the interview, some participants did say that deciding on a ‘future’ thing had been difficult, but nevertheless they all had chosen something. Towards the end of the interview I asked about their ‘things’ and we spoke about each object in turn, exploring why they had brought a particular object, how it related to their past/future, and whether and how this was something they used in their day-to-day lives.

Reflecting on the interviews I was wondering what made this particular exercise helpful for exploring and speaking about ‘futures’. Other scholars have successfully drawn on objects to study memories, but none have turned their attention to the potential of objects for studying futures. In the paper I argue that what makes the object-method productive is to do with materiality. More specifically, I argue that what makes this method unique is the combination of ‘materiality as method’ as well as the ‘materiality of the method’, and that this double materiality at play is what is producing elaborate future narratives. In other words, via the materiality of the objects, specific imagined futures become ‘within reach’ for participants, with the object serving as an anchor for these future narratives. The method suggests a temporal complexity as well: the future objects come to represent futures that the participants have already taken steps towards; they are ‘futures-already-in-the-making. Drawing on Jose Esteban Munoz, we can consider them ‘futures in the present’, that is, futures that already exist, perhaps just in glimpses, in the present.

To make this argument I draw on both narrative research, material culture studies and qualitative research methodology. One key source of inspiration was Liz Moor and Emma Uprichard’s work on material approaches to empirical research, where the authors argue for paying greater attention to the ‘latent messages’ of methods and data, for instance in the form of sensory and emotional responses but also, as I point to in the paper, the messages conveyed by a dirty and bent P plate and a carefully crafted name tag.   Due to limitations of space, the published paper focuses on the ‘future’ objects and the future narratives generated through these, and only briefly mentions the ‘past’ object that participants also brought to the interview. This is due to the paper’s ambition to highlight the potentials of using object methods, and a focus on materiality more generally, in research on futures. However, for a full analysis of the insights gained through this method, both in terms of the settled and unsettled future narratives and the normative dimensions shaping which objects became ‘proper’ objects for the interview situation, both ‘past’ and ‘future’ objects should be analysed together.

Read the full article in the IJSRM here.

featured, Notebook

Measuring Measures during a Pandemic

by Paul Romanowich & Qian Chen,

The spring 2020 semester started like many others before – frantically preparing class materials, finalizing research proposals, and trying to squeeze in one last getaway trip. However, by mid-March 2020 that normalcy had fallen by the wayside. Like it or not, classes were now all remote, disrupting both data collection and plans for any meaningful travel during the summer. But what about that data that was collected? Was it any good, considering what our participants were experiencing? Not surprisingly, little research has focused on the impact major environmental disruptions have on data reliability, given how rare and unpredictable those disruptions are (have you ever experienced a pandemic before 2020?!?). However, we were fortunate to be collecting repeated-measure impulsivity data throughout the spring 2020 semester. Thus, this research note focuses on whether data obtained in the immediate aftermath of the beginning of the COVID-19 pandemic is reliable, from a test-retest perspective.

Our original research question centered around whether decreasing one aspect of impulsivity, delay discounting, would have a positive effect on test scores for Electrical and Computer Engineering students. Like many personality traits, delay discounting rates have been shown to be relatively stable via test-retest data (i.e., trait-like). However, there is also a growing literature that episodic future thinking (EFT) can decrease delay discounting rates, and as a result decrease important impulse-related health behaviors (e.g., smoking, alcohol consumption, obesity). Thus, delay discounting also shows state-like properties. We hypothesized that decreasing delay discounting rates via EFT would also decrease impulse-related academic behaviors (e.g., procrastination), resulting in better quiz and test scores. To accurately measure temporal aspects of delay discounting, EFT, and class performance students completed up to 8 short (27-items) delay discounting tasks from January to May 2020. Multiple EFT trainings significantly decreased delay discounting rates relative to a control group (standardized episodic thinking – SET). However, the impact of EFT on academic performance was more modest.

Although the data did not support our original hypothesis, we did still have repeated-measure delay discounting data throughout the semester, which included data from March 2020 when classes were switched from in-person to fully remote. This repeated-measure data set up a series of Pearson correlations throughout the semester between delay discounting rates at two points in time (e.g., delay discounting rates at the beginning of the semester in January 2020 and end of the semester in May 2020). Importantly, students in the EFT group completed a delay discounting task on March 22, 2020 – 11 days after the official announcement that all classes would be fully remote for the remainder of the semester. In terms of test-retest reliability, the data collected on March 22, 2020 stood out as not like the other. Whereas delay discounting task test-retest reliability was high throughout the semester (supporting previous studies), most correlations using the March 22, 2020 data was nonsignificant, suggesting poor test-retest reliability. Thus, it appeared that the COVID-19 pandemic had significantly, but only temporarily, decreased test-retest reliability for delay discounting rates.

The EFT data also afforded us a way to look at changes more qualitatively in behavior before and after March 22, 2020. As a part of the EFT trainings, students came up with three plausible positive events that could happen in the next month, 6 months, and one year. We coded these events as either having COVID-19 content or not for all students. Predictably, events containing COVID-19 content did not appear until March 22, 2020. However, this event content changed as the semester progressed. On March 22, 2020, most (6 of 7 events) of the content was for the 1-month event. By May 7, 2020 only two students included COVID-19 content, and this was for the 6-month event. Thus, students were more concerned with COVID-19 in March 2020 and as a closer temporal disturbance, relative to May 2020. Perhaps this focus on COVID-19 in the near future disrupted delay discounting rates. We can’t be sure from this data, but the idea is intriguing.

Although this research note was not a rigorously controlled experiment to explicitly examine test-retest reliability for delay discounting, there are still some important points to take from the obtained data. First, it does appear that large environmental disruptions in participants life can significantly change test-retest reliability on standardized measures. Social and behavioral science researchers should be aware of this when interpreting their data. It may also be worthwhile to include a brief measure for significant life events that may be occurring concurrently with their participation in the task. Second, the change in test-retest reliability we observed was only temporary. This is actually good news for researchers, in that even significant environmental disruptions seem to have a minimal impact on test-retest reliability one month later. Perhaps we are more resilient as a species than we typically give ourselves credit for. Lastly, we have no doubt that other social and behavioral science researchers collected similar repeated-measure data throughout the spring 2020 semester. One way to be more confident that our results are not an outlier is through replication. Although we can’t (and don’t want to!) replay the beginning of the COVID-19 pandemic, researchers around the world could profitably begin to combine their data for specific well-validated measures to examine how this large environmental disruption may have systematically affected their results. The same could be done for other large environmental events, such as earthquakes or wars. The end result would be a better understanding of how these environmental disruptions impact those measurement tools that we base many of our theories and treatments off of.

Read the full article in IJSRM here.

featured, Notebook

Bringing “context” to the methodological forefront

By Ana Manzano & Joanne Greenhalgh,

In his latest methodological writings, Prof Ray Pawson (2020) noted that the Covid-19 pandemic:

 “covers everything from micro-biology to macro-economics and all individual and institutional layers in between”.

The current global pandemic could be considered the mother of all contexts. Many will conclude that we could not reduce the impact of Covid-19 in our lives to a limited number of contextual factors such as disease, bereavement, home working, schools closures, travel bans, etc. Covid-19 was and continues to be a force that impacts everything through a complex combination of omnipresent uncertainty, fears, risk management and materiality (masks , PCR tests, and hydrogenic gel). Our paper Understanding ‘context’ in realist evaluation and synthesis (Greenhalgh & Manzano, 2021), just published in the International Journal of Social Research Methodology, reflects precisely on how methodologically complex context is and reviews how context is conceptualized and utilized in current realist evaluation and synthesis investigations.

Perhaps, the most useful of all the quotes mentioned in our paper is one of French sociologist, Raymond Boudon’s (2014, p. 43) who reminds researchers that in the social sciences, it is impossible to talk about context in general terms, since context is always defined specifically:

The question as to “What is context?” has actually no general answer, but answers specifically adapted to the challenging macroscopic puzzles the sociologist wants to disentangle.

Context is somehow everything and, in some ways has become “nothing” with many methodological writings on causality focusing on the more attractive concept of “mechanisms”. Our paper projects context from its eternal background position in peer-reviewed papers, trials and research results, to the foreground.  Although context is a key concept in developing realist causal explanations, its conceptualisation has received comparatively less attention (with notable exceptions e.g. Coldwell (2019)). We conducted a review to explore how context is conceptualised within realist reviews and evaluations published during 2018. We purposively selected 40 studies to examine: How is context defined? And how is context operationalised in the findings? We identified two key ‘narratives’ in the way context was conceptualised and mobilized to produce causal explanations: 1) Context as observable features (space, place, people, things) that triggered or blocked the intervention; assuming that context operates at one moment in time and sets in motion a chain reaction of events. 2)  Context as the relational and dynamic features that shaped the mechanisms through which the intervention works; assuming that context operates in a dynamic, emergent way over time at multiple different levels of the social system. 

We acknowledge that the use of context in realist research is unlikely to be reduced to these two forms of usage only.  However, we argue that these two narratives characterise important distinctions that have different implications for the design, goals and impact of realist reviews and evaluations.  Seeing context as a ‘thing’, that is, as a  ‘feature that triggers’ suggests that one can identify and then reproduce these contextual features in order to optimise the implementation of the intervention as intended.  This reinforces a view that it is possible to isolate ‘ideal’ contexts that determine the success of an intervention. 

On the contrary, seeing context as a dynamic interaction between contexts and mechanisms implies that contexts are infinite, embedded and uncontrollable. Knowledge gained about how contexts and mechanisms interact can be used to understand how interventions might be targeted at broadly similar contextual conditions or adapted to fit with different contextual conditions.  This latter approach eschews the idea that there are ‘optimal’ contextual conditions but argues that successful implementation requires a process of matching and adapting interventions to different evolving circumstances. 

Our paper will disappoint those who seek a practical definition that will help the ever impossible task of distinguishing mechanisms from contexts in causal explanations. We have some sympathy with Dixon-Woods’ claim (2014, p. 98) about distinguishing mechanisms from contexts in realist studies:

 I am inclined towards the view that discussions of what constitutes a mechanism rapidly become unproductive (and tedious), and that it is often impossible, close up, to distinguish mechanism from context.

Since much methodological thinking focuses on mechanisms and funders are (typically, though not exclusively)  interested in outcomes, contexts are, if anything,  rather “annoying”. Context, with its symbiotic relationship with mechanisms,  confuses and distracts researchers in their most important search for mechanisms ‘holy grail’. Our paper demonstrates that the answer to that holy grail pursuit is precisely in that symbiotic relationship, in which contexts are relational and dynamic features that shape the mechanisms through which interventions work. Context operates in a dynamic, emergent way over time at multiple different levels of social systems.

Finally, we are mindful that, in Pawson’s own words when we discussed this paper with him, ‘context’ can mean ‘absolutelybloodyeverything’ and so it is very difficult to perceive that its usage in realist research is reduced to the two forms identified in our review.

Read the full IJSRM article here.

References

Boudon, R. (2014). What is context? KZfSS Kölner Zeitschrift für Soziologie und Sozialpsychologie 66 (1), 17-45

Coldwell, M. (2019). Reconsidering context: Six underlying features of context to improve learning from evaluation. Evaluation, 25 (1), 99-117.

Dixon-Woods, M. (2014). The problem of context in quality improvement. Perspectives on context. London: Health Foundation. 87-101

Greenhalgh, J. and Manzano, A. (2021) Understanding ‘context’ in realist evaluation and synthesis. International Journal of Social Research Methodology. https://doi.org/10.1080/13645579.2021.1918484 Pawson, R. (2020). The Coronavirus response: A realistic agenda for evaluation. RealismLeeds Webinar July 2020. https://realism.leeds.ac.uk/realismleeds-webinar-series/