To participate, you may either post a contribution to an existing discussion by selecting the thread for that topic (and then click on "Post Reply") or start a new thread by clicking on "New Topic" below.
The transition to Stage 2 of the deliberations is currently underway but will take some time to complete. In the meantime, we very much welcome additional contributions to the existing threads in this forum.
For instructions on how to follow a discussion thread by email, click here.
University of British Columbia
- Posts: 38
- Joined: Fri Feb 26, 2016 9:59 pm
Here are links to the policies of a few journals that say something in particular about transparency for qualitative research:
- APSR: The APSR's submission guidelines are here. And a set of FAQs about the APSR's transparency policy is here.
- The Italian Political Science Review provides one of the more specific statements about transparency for qualitative research. Scroll down to "Data access and research transparency in qualitative research."
- Conflict and Cooperation's policy, which discusses both qualitative and quantitative forms of evidence, is here (click on "Submission Guidelines" tab, then "Data" section).
- Research and Politics's policy is here
- Perspectives on Politics, while not a DA-RT signatory, has issued "Supplementary Materials and Data Guidelines" that separately discuss quantitative and qualitative research.
- European Political Science Review, also not a DA-RT signatory, states a transparency policy here under point 3.
- The editors of Comparative Political Studies have indicated that they want to hear from the QTD process before putting their transparency policy for qualitative research into place.
The Comparative Politics section newsletter issue that Hillel refers to is here.
Here is what I found.
1) The APSR was the only journal trying to spell out in some detail transparency guidelines. The other journals either explicitly reject them or show little dialogue with DA-RT related concerns.
2) The APSR guidelines for qualitative and historical/archival research are little more than re-articulations of current practices. Scholars working with documents are reminded to be careful with their footnotes and, if adventurous, maybe consider active citations. Quantitative scholars, in turn, are required to make their data more readily available. Pretty déja vue stuff.
3) The APSR guidelines were the vaguest for ethnographic or interview-based research. This combine with the greater explicitness for quantitative and archival research might contribute to the impression that research involving direct human subjects might be expected to adhere to practices of other research traditions.
4) The APSR guidelines were surprisingly vague and tentative in spelling out analytical transparency. If you think of analytical transparency (e.g. "clearly mapping the path from evidence to claims.") in terms of valid causal inference, then the question becomes how specific/strict do you want to get about analytical transparency? Is active citations the new gold standard? Or do you want to include Bayesian consideration of test strength? Or should scholars be expected to spell out the underlying ontological assumptions about their data (i.e. conditional independence, uniformity)? Analytical transparency could be a real Pandora's box where it is not clear how it differs from the key issues we commonly discuss under the labels methodology or epistemology.
University of Pennsylvania
- Posts: 1
- Joined: Thu Apr 07, 2016 9:04 pm
Presumably the answer is that it is difficult enough to recruit reviewers even if they are not required to make transparently available the inferential and data bases for their recommendations. From a practical point of view, in other words, editors might say they simply cannot function unless they excuse reviewers from the process that they nevertheless claim is necessary for the credibility of the scientific progress. OK, but if there is room for practical compromise so that scientific work can proceed, then there is no room for an absolutist imposition of DA-RT standards since the editors are already conceding that scientific judgments can be made, and can be acted upon as credible, without honoring those standards.
Univ of New Mexico
- Posts: 16
- Joined: Wed Apr 06, 2016 9:20 am
University of Massachusetts, Amherst
- Posts: 2
- Joined: Wed May 18, 2016 9:20 am
Also, I very much appreciate Ian's points. Even outside of the DART debate, I've always been surprised by the weak evidentiary standards for reviewers.
I am not sure I get Ian Lusticks's point right, which is a valuable point (if I am getting it right): I guess we all agree peer review does not work perfectly. But I am not sure what the reviewers should be transparent about. In my reviews, I rarely make claims that needed to be verified by DA-RT standards. If a theoretical argument is bad, then it is a theoretical point. If the choice of cases is not discussed, then I criticize this and I usually give the part in the manuscript to which I refer. If I review a QCA study, it might happen that I try to reproduce the result if I have access to the data. If I find something in the analysis I consider weak, I submit my analysis to the editors and do not simply make a claim in my review.
In any case, the shortcomings of the peer review system should not be turned against DA-RT. If at all, we might use DA-RT to make peer review better. Transparency of the peer review process is an important issue and making the reviews publicly available is one of the issues that is regularly raised in this debate.