Qualitative Transparency Deliberations

[Paolo Spada Southampton]

While QTD is a discussion that aims to be broader than the discussion around DART and JETS, I think it is still important that part of the discussion engages DART and JETS because people are coming to these boards in part motivated by these changes and thanks to the protest movement that highlighted the discussion. Thus I want to add some of my thoughts about QTD insofar and what maybe could be helpful going forward with respect constructing together a proposal to make JETS clear and operational that journals can adopt and some changes in the wording of JETS when required.

I think that the discussion around JETS has insofar well captured the extreme 'realizations' of the possible scenarios and realities facilitated by JETS. On one hand QTD has clearly captured a variety of legitimate fears regarding potential extreme consequences of JETS, on the other the article of Elman and Lupia in the CP newsletter (here) provides a "nothing will change" counter message that combined with some of the older discussions around active citations as precondition for data replication (here) instead of engaging fears just dismisses them as miss-construed miss-understanding or reactions from luddites that have something to hide or do not know anything about 'good' methodology.

This is fairly normal given that QTD offered the first space for open discussion and thus vented all the things that had been badly compressed before. The discussion that preceded QTD simply did not do enough to engage the public. QTD with zero funding and a tiny team of volunteers has reached more than a 1500 people, many not in R1 universities, proving that the claim that nobody wants to discuss these things is empirically wrong.

For JETS purposes we have mapped well the extremes, now I think we should focus on the things the lie in between or we risk to not offer any operational solution.

For example, and the following are just 4 examples:

1) Active citations
what if active citations did not require to scan anything, but simply allowed people to have more space for a citation? When I cite my interviews, or a document that I think is important for my argument, I am always super-constrained by the word limit of the paper. My citations end-up being a one liner. If I had a hyperlink in which I could cite a larger portion of the document or interview that would be quite useful. Maybe I could even have an entire page to explain the context in case the point I am making is not linked to a specific smoking gun, but a series of interviews. This approach envisions the hypertext of an active citation as a "peak under the hood" additional space for explaining things for the curious reader. These hyperlinked texts would be totally under the control of the author and would have nothing to do with a data replication effort that as many colleagues have argued in QTD is difficult to apply as soon as we try to be a bit more nuanced. So my point is can we simply transform active citations in a space for more explanations that does not require any mandatory costly scanning/uploading? I think we can, and the discussion on this board about the meaty footnote (if I am not miss-understanding see here) goes in such direction and I think it is an example of operational suggestions that is not trapped in the extremes that exemplifies how I would recommend we go forward to build together a proposal to operationalize/amend JETS.

2) Human subject protection
I think that nobody in their right mind would require to endanger human subjects and nobody would ask to violate IRB compliance. I think that all editors agree with this and it is just an unfortunate wording of JETS that makes things unclear. We should propose a simple amendment that makes things clear. My personal sense is that this crucial heated debate stems from a miss-understanding so it is probably the easiest to solve on one level, on the other it will require a change in JETS wording, and given how defensive the proponents of JETS have been insofar it might not be as easy and it might require another petition or protest. Going forward I think that one operational frame could be "we all agree that human subjects come first, can we add a line in JETS that highlight such concept? It's a bit vague and we do not want to send a vague message on such important subject" or something like that. I know that there have been previous attempts to do such thing and I am proposing to just do another one. But my point is that such attempt should come from the understanding that nobody wants to endanger human subjects and it is a shared value, so it is just the vagueness we need to resolve not some profound ethical and divisive debate. Maybe we do not need to change the wording of JETS, but we just need to ask the journals to add a strong operationalization of such rules in their bylaws.

3) Editors role & engaging more the editors
On one hand it is obvious that the editor of a journal has complete control on what gets published. On the other why reify such obvious operational thing in JETS? What is the reason to do that? This is one of the things I do not understand of JETS and I would like an explanation. Going forward I think we need to involve as many editors as possible in this conversation. I have noticed one or two posts from editors. Maybe we should have a working group dedicated to editors and asking questions to editors. They are not some strange aliens, they are colleagues and many must be as confused as we are about JETS.

4) Clarifying once and for all that there is no mandatory added costs for qualitative research
A lot of people in these boards have repeated that their research would be impossible due to added costs. My sense is that nobody wants that and actually journals would like more qualitative research being produced and have a hard time getting more. At the same time JETS is so vague that the fears of our colleagues should not be dismissed and we need a clear understanding about what will be required operationally. So this links with point 3, we need to talk to editors, without such exchange we risk to be trapped in a non operational discussion. I do not think any editor would require uploading my badly scrabbled irrelevant field note, all the transcripts of my interviews etc. etc. So what would they require in practice?

In conclusion, I propose we build 1 working group that will focus on engaging the editors and operationalizing/amending JETS. This way in all the other working groups we can focus on more relevant and interesting discussions.

[Tim Buthe HfP/Technical Univ of Munich & Duke University]

Thank you, Paolo. These are very constructive suggestions. I particularly like the one regarding the use of hyperlinks that would not just lead to primary sources or other documents but allow elaboration and background information beyond the word limit. This could be a very useful alternative to expanding the word limit (which I suspect few journals may be willing to do).

As to a clarification of the relationship between JETS and human subjects protection, there was a proposal with quite specific language that seemed to achieve just such a clarification, which was drawn up and sent to JETS editors with support from both a large group of scholars who've been rather critical toward DART *and* the DART leaders. And it still failed to gain unanimous acceptance from the JETS editors, which apparently is required to bring about a change in JETS since that statement was drawn up without any institutional or governance structure attached to it (quite stunning, if you think about it, given that it was the product of the intensive work by a group of political scientists). Some of the journals individually, however, have made their commitment to the primacy of human subjects protection quite clear.

[Guest]

It is interesting that you think that a unanimity requirement to change a voluntary consensual agreement is not a governance structure. If the next time you enter into such an agreement (a sales contract, for example), you allow a subset of the signatories to unilaterally change the terms of the agreement, the people who depend on you for sustenance will be very disappointed, as your assets will soon be taken from you. Requiring unanimity for changes to voluntary agreements is not an unusual governance structure in the west, it is a norm.

[Tim Buthe HfP/Technical Univ of Munich & Duke University]

Fair enough; thanks for pointing this out. My statement about the lack of a governance structure was based on the possibly erroneous assumption that the lack of an explicit statement about a mechanism or procedure for amendments or revisions in the JETS statement (and the accompanying documents) had by accidental implication made unanimity the decisionmaking rule for revisions--and that the JETS statement was in this sense without a governance structure. It is certainly correct that allowing changes only by unanimity is a kind of governance structure, and it is possible that a unanimity rule was indeed consciously and intentionally adopted (though I would not want the recognition of this possibility to seen as suggesting that it was so; I simply don't know, and given that the poster did not identify him/herself, we have no way of telling whether s/he knows or simply speculates).

Having spent a large part of the past 15 years or so studying standard-setting processes and organizations (and JETS clearly is an attempt to set standards), I think I can confidently say that unanimity is *not* the norm of standard-setting organizations--and where it is, it seems to be a conscious attempt to write a particular solution into stone, i.e., bind the hands of successors through institutionalization, a political strategy about which Terry Moe has written on several occasions, including very eloquently in Perspectives on Politics in June 2005 (vol.3 no.2, pp.215-233). Nor does it appear to be the norm in standard-setting bodies to leave unaddressed the question of how a standard can/should be amended, to be governed only by general norms of contracting.

Also, if the original signatories indeed intended the JETS as something like a contract that was consciously intended to be changeable only by unanimity of all the signatories, the opening of the statement to any number of additional (journal editor) signatories implies an intent to make it successively harder to ever change the statement, as changes then require not just unanimity among the original x editors but unanimity among x+n editors, where n, the number of additional signatories is presumably increasing over time. This would imply a conviction in the supreme wisdom and finality of the JETS standards that few standard-setting bodies dare to proclaim for their "technical" experts--or an intent to create a self-undermining institution since (unlike in most contracts) there is nothing in JETS that precludes any editor or journal from un-signing the statement unilaterally and if the lack of adaptability makes an institution more and more dysfunctional, we generally expect this to increase the attractiveness of exit. Since the signatories include numerous thoughtful colleagues whose good judgment I very much trust, I strongly suspect that they did *not* in fact look upon this as a contractual agreement of the type suggested by the 5/28, 10:47pm post. But I readily grant that this is an inference, only--for which I hope to have now laid out my methods quite transparently... Also, just so that it doesn't get lost: The point of the second paragraph of my original post was that there has already been a proposal for the kind of amendment Paolo suggested (re. human subjects protection), but that it appears to be very difficult to get any such amendment adopted, given the apparent unanimity rule for changes, but that a number of JETS journals individually have (in my reading) been very responsive to the concern that the primacy of human subjects protection may need to be affirmed.