Page 1 of 1

A Brighter Side to Data Dissemination

Posted: Mon May 02, 2016 5:14 pm
by evanlieb
I have read with interest the comments posted on this site (and kudos to Alan Jacobs and Tim Buthe for doing such an excellent job promoting thoughtful online engagement.) Within this forum, in the CP newsletter, and in private conversations, a range of colleagues have raised important concerns that point to quite serious limitations to the DART and JETS initiatives and the potential expectations and demands that those initiatives imply.

Virtually by definition, qualitative research is eclectic, and observations are not easily ordered into rows and columns, which makes dissemination — an operational implication of “research transparency” — potentially challenging. Most observations are recorded in the form of transcripts, which in turn generate privacy concerns that are not as easily protected as in the case of sensitive, individual-level quantitative data, based on closed-ended questionnaires. When people talk, they reveal personal and self-identifying information. As a result, many contributors to this forum highlight that there will be no easy qualitative analog to the archiving of a quantitative dataset + codebook + code. And as a scholarly community, we clearly need more time to test the waters with best practices for dissemination.

That said, I fear that the discussion has been disproportionately negative with respect to the likely consequences of specific initiatives towards greater “transparency” or “explicitness” (hat tip to Craig Parsons for the latter) that involve dissemination of data, protocols, transcripts, archives, or other materials that provide a view of how research was conducted and the data that were generated in a form that would not be appropriate for publication within the context of a journal article.

I think a more balanced discussion needs to consider not just the downsides, but also the possible scholarly and professional gains of incentivizing greater dissemination of such materials for individual qualitative/mixed-method scholars: Organization of data and research in a manner that is amenable for sharing is likely to force scholars to be more self-conscious about the quality of those data and how they are analyzed; Such data are more likely to become used and cited public goods, providing attention to individual scholars and their scholarship; such dissemination may, in some cases, increase readers’ interest in and confidence in the soundness of the findings. Particularly for graduate students and junior scholars, the dissemination of additional materials could lead to much greater attention to their work, engagements in interesting collaborations, etc., as is frequently the case among quantitatively-oriented scholars when they share their data. Is this potentially “extra” work, and might it be difficult to do? Yes. But if done well, with careful ethical foresight and oversight, I think it could be rewarded in terms of recognized quality of research, citations, etc.

What should those practices be? That is a discussion I wish we were having. At the very least, greater attention ought to be paid to the efforts of contributions such as Elman, Kapiszewski, and Vinuela (2010) and Moravcsik (2010, 2014), which seem to me to be positive efforts to make dissemination relatively straightforward. As has been well noted, many types of scholarship may not fit well with these strategies. On the other hand, much qualitative research does not involve highly sensitive individual-level materials and might benefit from the use of such tools. The objection that these strategies will not work for everyone does not mean that they won’t work for anyone. Just as high-quality qualitative research demands creativity and recognition of context, so too should the development of strategies for sharing such work.

To be clear, there are clearly costs and risks associated with increasing the norms around “research transparency.” But to view dissemination as merely a “response” to suspicious reviewers I think loses sight of the rich scholarly contribution that such practice might entail. As a policy, I would rather see data dissemination be an encouraged “opt in” option (thanks to Regina Bateson and Kathy Thelen for sharing important views on this), rather than a mandatory practice that would force some scholars to explicitly “opt out.” We should reward scholars who provide clear and well organized public goods in the form of supplementary information. But we should not punish those who feel that such dissemination is impossible or unnecessary beyond what they report in their main published work. Norms and standards should develop more organically within research communities — and readers and reviewers will either come to demand a certain degree of dissemination or they will not.

There is a potentially brighter side to the practices associated with increased and explicit attention to “research transparency,” and while I am against mandating any particular practices at this moment, I am certainly in favor of encouraging efforts and experimentation.

Re: A Brighter Side to Data Dissemination

Posted: Tue May 03, 2016 9:29 pm
by Marcus Kreuzer
I like Evan's emphasis that qualitative transparency standards are still emerging and thus will require experimentation. I think experimentation could easily be extended beyond data dissemination. It would be particularly encouraging if journals themselves would lead the way with several possible experiments:
1. Facilitate post publication online discussion as is the norm in many of the natural sciences.
2. Be more welcoming to submissions from different stages of the research cycle, that is, not to privilege the standard hypothesis testing article. A point Evan emphasized in a recent conference paper.
3. Provide technology for active citations.
4. Maybe even bring back the footnote rather than hiding transparency comments bound for footnotes in endnotes.
5. Don't count footnotes towards the overall word count. This would remove a powerful disincentive to include transparency commentary.
6. Place greater value on literature reviews, especially the ones where the author carefully and thoughtfully cross-examines competing explanations. This strikes me as a key element of analytical transparency.
Such experiments might create incentives for scholars to experiment to see what works and ultimately transform transparency criteria through practice rather than by editorial fiat.