Substantive Dimensions of the Deliberations

Forum rules

We encourage contributors to the Discussion Board to publicly identify by registering and logging in prior to posting. However, if you prefer, you may post anonymously (i.e. without having your post be attributed to you) by posting without logging in. Anonymous posts will display only after a delay to allow for administrator review. Contributors agree to the QTD Terms of Use.

Instructions
To participate, you may either post a contribution to an existing discussion by selecting the thread for that topic (and then click on "Post Reply") or start a new thread by clicking on "New Topic" below.

The transition to Stage 2 of the deliberations is currently underway but will take some time to complete. In the meantime, we very much welcome additional contributions to the existing threads in this forum.

For instructions on how to follow a discussion thread by email, click here.

hillelsoifer

Journal standards in practice

PostWed Apr 20, 2016 9:12 am

I wonder whether it might be useful to discuss the actual policies put in place by journals in response to DA-RT? I personally found the statements of the editors of the APSR and CPS in the CP newsletter to diverge sharply from the fears raised by many on this board, and think that they might provide groundwork for scholars to respond to knee-jerk criticisms of qualitative work without falling prey to many of the concerns raised in this conversation. I should note, however, that my work is largely archival, and mixes qualitative and quantitative work, and thus may not have to confront the same issues raised by other scholars posting here who work with human subjects or in more immersive modes of research. I'd be interested to hear, therefore, what others might think about the standards in practice.

Hillel Soifer

Post Reply

Alan Jacobs
University of British Columbia
Posts: 38
Joined: Fri Feb 26, 2016 9:59 pm

[Steering Committee] Journal standards in practice

PostWed Apr 20, 2016 6:48 pm

Thanks very much for this comment, Hillel. It would indeed be helpful to hear what people think of those journal policies that are currently in place. Judging by their online author guidelines, many policies of JETS journals remain rather general or appear to still be under development. Some indicate specific rules for quantitative research, but not for qualitative. But what do people think of the policies for qualitative research that are have been posted so far?

Here are links to the policies of a few journals that say something in particular about transparency for qualitative research:

- APSR: The APSR's submission guidelines are here. And a set of FAQs about the APSR's transparency policy is here.

- The Italian Political Science Review provides one of the more specific statements about transparency for qualitative research. Scroll down to "Data access and research transparency in qualitative research."

- Conflict and Cooperation's policy, which discusses both qualitative and quantitative forms of evidence, is here (click on "Submission Guidelines" tab, then "Data" section).

- Research and Politics's policy is here

- Perspectives on Politics, while not a DA-RT signatory, has issued "Supplementary Materials and Data Guidelines" that separately discuss quantitative and qualitative research.

- European Political Science Review, also not a DA-RT signatory, states a transparency policy here under point 3.

- The editors of Comparative Political Studies have indicated that they want to hear from the QTD process before putting their transparency policy for qualitative research into place.

The Comparative Politics section newsletter issue that Hillel refers to is here.

Post Reply

Guest

Re: Journal standards in practice

PostSat Apr 23, 2016 5:44 pm

A number of posts extended invitations to either make specific transparency recommendations or to comment on existing ones. So, I looked at the guidelines of the journals that Alan kindly listed above.
Here is what I found.
1) The APSR was the only journal trying to spell out in some detail transparency guidelines. The other journals either explicitly reject them or show little dialogue with DA-RT related concerns.
2) The APSR guidelines for qualitative and historical/archival research are little more than re-articulations of current practices. Scholars working with documents are reminded to be careful with their footnotes and, if adventurous, maybe consider active citations. Quantitative scholars, in turn, are required to make their data more readily available. Pretty déja vue stuff.
3) The APSR guidelines were the vaguest for ethnographic or interview-based research. This combine with the greater explicitness for quantitative and archival research might contribute to the impression that research involving direct human subjects might be expected to adhere to practices of other research traditions.
4) The APSR guidelines were surprisingly vague and tentative in spelling out analytical transparency. If you think of analytical transparency (e.g. "clearly mapping the path from evidence to claims.") in terms of valid causal inference, then the question becomes how specific/strict do you want to get about analytical transparency? Is active citations the new gold standard? Or do you want to include Bayesian consideration of test strength? Or should scholars be expected to spell out the underlying ontological assumptions about their data (i.e. conditional independence, uniformity)? Analytical transparency could be a real Pandora's box where it is not clear how it differs from the key issues we commonly discuss under the labels methodology or epistemology.

Post Reply

Ian Lustick
University of Pennsylvania
Posts: 1
Joined: Thu Apr 07, 2016 9:04 pm

Re: Journal standards in practice

PostSun May 15, 2016 3:35 pm

I'd like to raise an infinite regress question. If editors are unwilling to assign credibility to judgments offered by the scholars who submit manuscripts for publication unless those scholars meet requirements for transparency and replicability, then how can those editors assign credibility to the scholars who offer positive or negative judgments about those manuscripts in the peer review process unless each reviewer's review is also subject to those same standards of transparency and replicability? How many of us, as manuscript reviewers, have seethed at the validation and verification errors of reviewers whose errant and discreditable judgments often drive editorial decisions? Why isn't it true, from the DA-RT perspective, that what is good for the manuscript submitter is good for the manuscript reviewer?

Presumably the answer is that it is difficult enough to recruit reviewers even if they are not required to make transparently available the inferential and data bases for their recommendations. From a practical point of view, in other words, editors might say they simply cannot function unless they excuse reviewers from the process that they nevertheless claim is necessary for the credibility of the scientific progress. OK, but if there is room for practical compromise so that scientific work can proceed, then there is no room for an absolutist imposition of DA-RT standards since the editors are already conceding that scientific judgments can be made, and can be acted upon as credible, without honoring those standards.

Ian Lustick

Post Reply

Mala Htun
Univ of New Mexico
Posts: 16
Joined: Wed Apr 06, 2016 9:20 am

Re: Journal standards in practice

PostSun May 15, 2016 11:03 pm

[quote="ilustick"]If editors are unwilling to assign credibility to judgments offered by the scholars who submit manuscripts for publication unless those scholars meet requirements for transparency and replicability, then how can those editors assign credibility to the scholars who offer positive or negative judgments about those manuscripts in the peer review process unless each reviewer's review is also subject to those same standards of transparency and replicability?" "if there is room for practical compromise so that scientific work can proceed, then there is no room for an absolutist imposition of DA-RT standards since the editors are already conceding that scientific judgments can be made, and can be acted upon as credible, without honoring those standards."

EXCELLENT POINT!!!!

Post Reply

Guest

Re: Journal standards in practice

PostTue May 17, 2016 10:19 am

Very much agree with Ian Lustick's point.

Post Reply

Amel Ahmed
University of Massachusetts, Amherst
Posts: 2
Joined: Wed May 18, 2016 9:20 am

Re: Journal standards in practice

PostThu May 19, 2016 3:32 pm

I think part of the problem is that there are a lot of conflicting statements. I've been following this debate from the beginning (though probably not as closely as I should have) and I still don't really have a handle on what is proposed. Figuring out the requirements for archival work for example – yes, most of the journals simply say that we need clear citations identifying the archive, location, etc., but then the original APSA statement seems to suggest that we would need to make the original source material available so as not to place too great a burden on those who would want to track it down. So is that still on the table? If so, that would be ludicrous for most archival research. How would I make my microfilm and rare documents available? Then there are the AJPS badges, which make no sense for most qualitative work, esp. archival. The point it, there are a lot of different statements out there, many of them conflicting. I think you're right Hillel that so far the statements seem rather benign, but I suspect the journals have been somewhat tentative about their policy statements, in part due to the push back. But those policies are likely to evolve and it is in our interest to make sure they evolve in a way that makes sense.

Also, I very much appreciate Ian's points. Even outside of the DART debate, I've always been surprised by the weak evidentiary standards for reviewers.

Post Reply

Ingo Rohlfing

Re: Journal standards in practice

PostTue May 24, 2016 6:04 am

Ingo Rohlfing, University of Bremen:
I am not sure I get Ian Lusticks's point right, which is a valuable point (if I am getting it right): I guess we all agree peer review does not work perfectly. But I am not sure what the reviewers should be transparent about. In my reviews, I rarely make claims that needed to be verified by DA-RT standards. If a theoretical argument is bad, then it is a theoretical point. If the choice of cases is not discussed, then I criticize this and I usually give the part in the manuscript to which I refer. If I review a QCA study, it might happen that I try to reproduce the result if I have access to the data. If I find something in the analysis I consider weak, I submit my analysis to the editors and do not simply make a claim in my review.
In any case, the shortcomings of the peer review system should not be turned against DA-RT. If at all, we might use DA-RT to make peer review better. Transparency of the peer review process is an important issue and making the reviews publicly available is one of the issues that is regularly raised in this debate.

Post Reply