In this blog, IJSRM editors Ros Edwards and Brian Castellani share views on the issue of basic data sharing and potential implications for the International Journal of Social Research Methodology.
ROS: Our publisher has adopted a Basic Data Sharing Policy where they state that journals that they publish will encourage authors to share or make open the data underlying their article publication where this doesn’t cut across privacy and security of human subjects. This is a requirement that’s coming to the fore across disciplines, research funders and the academic publishing world. The data set to be shared or made open by the author/s should be the data needed for independent verification of research results in the article. The idea is that authors sharing data supports research transparency, reproducibility and replicability, and enables researchers to build on others’ work.
While I generally support ideas about transparency and data sharing for research reuse, I wonder about its applicability and implications for our journal given our focus on methodology. This is especially the case from the perspective of a qualitative methods article.
I’ll start us off with my first concern. The idea of data sharing for replication and verification is based on the notion that data is free-standing and that anyone analysing the same data set will come to the same results – or different ones if the analysis has not been carried out correctly. I’m not sure this is the case for quantitative data in the social sciences, and I’m certain that it isn’t the case for qualitative research. For example, given a single qualitative data set, a researcher with a psychosocial analytic approach may come up with quite different research findings to one who approaches the data from a critical realist or interpretive perspective. Neither will be ‘wrong’; they are providing different angles on the data. So, the researcher and their epistemological standpoint is part of the method of analysis. That isn’t necessarily an argument against data sharing of course – as I say, I can see the benefit of making data available for other researchers to use. It’s just that some of the rationale for publishers requiring data sharing for articles that are published in their journals needs to start from a different basis.
BRIAN: I agree with much of what you say, Ros. I do not pretend to be an expert in all aspects of the discussion, but the problem for me is what the motivation is behind it all, which seems to have a lot to do with the recent debates within psychology and, more widely, social science regarding the ‘reproducibility crisis.’
For a good summary of the debate and its historical errors, I recommend reading, Daniele Fanelli’s “Is science really facing a reproducibility crisis, and do we need it to?” At the end of his article, he states:
To summarize, an expanding metaresearch literature suggests that science—while undoubtedly facing old and new challenges—cannot be said to be undergoing a “reproducibility crisis,” at least not in the sense that it is no longer reliable due to a pervasive and growing problem with findings that are fabricated, falsified, biased, underpowered, selected, and irreproducible. While these problems certainly exist and need to be tackled, evidence does not suggest that they undermine the scientific enterprise as a whole. Science always was and always will be a struggle to produce knowledge for the benefit of all of humanity against the cognitive and moral limitations of individual human beings, including the limitations of scientists themselves.
For me, it is not so much if the crisis exists, as much as what it says, methodologically speaking, about where most conventional social science remains presently, which is caught up in some epistemological variant of a positivist, quantitative methods based view of science and an emphasis on causality. And this is despite all the significant methodological critique and advance made in such fields as feminist methods, intersectionality theory, complexity science methodology, narrative inquiry, qualitative methods, mixed-methods, and the past several decades of the sociology and philosophy of science.
It also reminds me of Mike Savage’s (2009) article, Contemporary sociology and the challenge of descriptive assemblage (European Journal of Social Theory) in which he argues for the importance of description as a legitimate methodological and substantive goal of social science, and not just the development of models of causality.
Still, I do agree that sharing data is a great thing. Mainly because I also agree that, regarding issues such as policy evaluation and decisions on such things as health and mental health, etc, we need to be methodologically rigorous because people’s lives are dependent upon us doing good research. I also agree that there is pressure to produce increasingly novel results, and that scholars are often less motivated to examine further the results of their colleagues’ work. And I agree that one must always be worried about the influence that industry, politics, business and funding have on the results at which scientists arrive. But, I also agree that, one study does not a new insight make. Instead, it comes from a field of study and its key debates. I am also very aware that science advances, albeit in a very messy way, through debate and conflict.
In other words, while any given study might be brilliantly insightful, the world is too complex to be contained within the results of a single study or frame of thinking, and so all research will be undetermined by its evidence. And, for me, why this periodically suddenly surprises people I am not sure?
As such, and to repeat the point, I worry as you do about the need to share being more about a certain view of what, methodologically speaking, social science is able to do and how we respond in light of its inability to live up to that perception. And less about increasing the capacity of researchers and citizen scientists being able to access the data upon which a study is based.
ROS: Turning to my second, and I think more crucial point in terms of implications for whether or not IJSRM implements a data sharing policy then. In a methodology journal like ours, the articles that we publish are often researchers reflecting on their methods. For example, they may write about their negotiations around access to a setting, or about shifting power relations in designing participatory research, or about non-verbal communication in interviews. In a sense then, in these discussions the researcher is the data, or at least is a strong feature of it. The researcher is making the argument transparent through the article – arguably publishing the account is the transparent and sharing element. To require authors of such articles to publish their fieldnotes or interviews or whatever would feed my first point above, about data as free-standing, in an entirely inappropriate fashion.
BRIAN: This is an excellent point, Ros, and one that may actually function as a useful counterpoint to the first. In other words, for sake of argument, let’s assume there is a legitimate crisis in causality and reproducibility in science and a need to share data. It seems, then, that a useful medicine is a journal such as ours, where scholars are given the chance to reflect and think through their work or the work of others and the fields to which they belong to think how best, methodologically speaking, to proceed. In such instances, the actual data upon which their studies are based may sometimes be less important, given that, as you say, the goal of the article is transparency. Again, I don’t think a blanket statement of doing things one way or another is useful. But, then, that is the point, isn’t it?