Qualitative Transparency Deliberations

[John Gerring University of Texas at Austin]

Having read some of the previous posts - and paid some attention to the debate carried out in other venues over DA*RT - i have lots of thoughts, but let me try to boil things down so i don't take up too much of everyone's time...

Insofar as ours is an empirical endeavor the goal of transparency would seem to matter greatly. If an argument is based on some sort of evidence (be it interviews, observations, texts, pictures, shards, whatever) then it is a problem if the author cannot produce the evidence, as well as the form of analysis the researcher used to reach his/her conclusions. I grant that it is not always possible to do this, and i'll discuss some of these problems below. But my point is (I think) a simple one: transparency is good, ceteris paribus.

The debate over DA*RT rests on the ceteris paribus conditions. What sacrifices are entailed, and are they worth the costs (or in what situations might they be worth the cost)?

In answering this question there seems to be a common view that the obstacles are greater for quals than for quants. I agree, but only to a point. Let me give some examples...

1. Protecting sources. This is a persistent - and very real - worry about transparency. But it is certainly not unique to qual work; indeed, there are elaborate procedures for protecting the anonymity of survey respondents. Nor does it apply to all qual work, e.g., to research based on archives or published work (as Lisa Wedeen notes). Protecting sources, in my view, runs orthogonal to the qual/quant divide.

2. Costs for the researcher (time and money, if RA's are involved). Again, this is a persistent and very real concern. But it is certainly not unique to qual research. To be sure, it may be somewhat more time-consuming for qual researchers to prepare their notes (or other material) for archiving, including anonymizing sources, where necessary. But this is a matter of degrees, i would think. And the process is eased (i would hope) by the Qualitative Data Repository hosted at IQMR.

3. Proprietarial material. I can think of at least one obstacle to transparency that is found more commonly on the quant side than the qual side. Sometimes data used for research is owned by someone else. While it may be used, perhaps in an aggregate format, the owner of the data does not consent to make the data public. (I imagine there are examples of this in qual work but i can't think of any off the top of my head.) An example in the quant world is the influential World Bank Governance Indicators, developed by Daniel Kaufmann and collaborators. The data used to construct these indices is mostly proprietarial and therefore is not released with each version of the dataset. People are troubled by this - as they should be - but that doesn't seem to prevent publications in (top) scholarly journals.

Other examples might be discussed but i think the point is clear. There are costs to achieving greater research transparency, and they fall on all researchers. Perhaps the burden is somewhat heavier on the qual side, but we ought not imagine that the quantoids are getting off lightly.

By way of conclusion, i would suggest that we focus on the specific methodological issues that are raised by DA*RT rather than whether the work under review is quant or qual. It is perfectly reasonable to insist that no transparency initiative should compromise the confidentiality of sources, for example. But it does not seem reasonable to say that qualitative work should be exempt from the DA*RT, as some contributors to this symposium seem to suggest.

For those who are unhappy with standards adopted by the APSR or other journals, or by APSA, i would issue a challenge: let's work to improve them.

[Mala Htun Univ of New Mexico]

you should check out the other thread about the costs of compliance with data access requirements. for work based on interviews, this would involve transcription and anonymization, which is time consuming and expensive. on that thread,several folks pointed out that the costs could be prohibitive and that they personally would face huge barriers to continuing with qualitative work. i dont think we want to make a comparison with quant work (totally agree there are costs there too!) so much as to discuss how to manage the compliance costs. for me, this is important for its effects on the incentive structures underlying various forms of inquiry in the profession. see also my essay in the CP newsletter, just released last week. thanks! mala

[Jonathan Fox School of International Service, American University]

I second Mala Htun's about the compliance costs, especially for those do numerous informal interviews as part of full immersion in ethnographic field research.

The "let's work to improve this" spirit is what seems to drive this deliberative process, though insofar as this is unfolding well behind the decisions already made by influential journals, the playing field is less than level.

[Tim Buthe HfP/Technical Univ of Munich & Duke University]

Thank you to John, Mala, and Jonathan, for your contributions to this thread. I appears that you all agree (as do probably many others) that we do not want to reify the qualitative-quantitative divide--and that there are numerous different kinds of "qualitative" scholarship. Next week, we will start to move from the at-large deliberations of stage 1 of the QTD to stage 2. During stage 2, all of the original threads remain available, but the deliberations are supposed to continue, primarily, in 12-15 separate working groups, so as to allow more differentiated discussions of research transparency in specific qualitative research traditions.

All of this raises two important and now increasingly urgent question:

(1) For the kind of research you do, what are the aspects of research transparency that are most in need of differentiated deliberations during the 2nd stage of the QTD process? (Please be sure to specify the "kind of research you do")

And more generally: (2) How should the stage 2 working groups be delineated or defined?

Please feel free to weigh in here or by posting a reply on the general thread about how the working groups should be defined.

[robert mickey university of michigan--ann arbor]

Transparency is a great idea, and we should embrace much of this movement. I appreciate what John Gerring is up to and what he says here. But I don't think Andy Moravcsik's claim that qual work is in crisis because of transparency and replicability is a persuasive one. Qual work is considered by many scholars as unscientific largely for reasons having to do with number of cases, degrees of freedom, and the (perceived) absence of (good) rules/norms about the specification of causal claims and provision of evidence about them. Telling them where our sources come from won't matter. So I support qualitative researchers' adaptation of this movement on terms that make sense for qual. work, but I think the motivation should be a principled commitment to transparency, rather than the hope that "now maybe they'll like us." That's not going to happen short of a much larger transformation of qualitative political science (or of mainstream scholars' understanding of what science is).

[Fora Admin Qualitative Transparency Deliberations]

Transparency is a great idea, and we should embrace much of this movement. I appreciate what John Gerring is up to and what he says here. But I don't think Andy Moravcsik's claim that qual work is in crisis because of transparency and replicability is a persuasive one. Qual work is considered by many scholars as unscientific largely for reasons having to do with number of cases, degrees of freedom, and the (perceived) absence of (good) rules/norms about the specification of causal claims and provision of evidence about them. Telling them where our sources come from won't matter. So I support qualitative researchers' adaptation of this movement on terms that make sense for qual. work, but I think the motivation should be a principled commitment to transparency, rather than the hope that "now maybe they'll like us." That's not going to happen short of a much larger transformation of qualitative political science (or of mainstream scholars' understanding of what science is).

This is a test post.