Qualitative Transparency Deliberations

[Kurt Weyland Univ Texas - Austin]

Because transparency requirements would impose enormous logistical burdens and costs on qualitative researchers, and very disproportionately on us, I oppose these initiatives. All these efforts would create even more disincentives against qualitative research, and that's precisely what our "discipline" should avoid. Best, Kurt Weyland, Univ. Texas-Austin

[Guest]

I agree completely for these and other reasons. What seemed at first a benign attempt to get quantitative scholars to make their data available sooner rather than later has morphed into a disciplining move (in the Foucaultian sense) that would lead qualitative scholars to avoid (if not outright boycott) a slew of otherwise-valuable journals. The long term negative impact would be huge.

[Sheena Greitens University of Missouri]

I'd like to add my agreement here. The costs are both large and disproportionately distributed, and the incentives align to seriously discourage qualitative research. What began as a very sensible suggestion that qualitative researchers consider how best to make their methods more transparent to increase the credibility of their work has morphed into a set of requirements that seem inappropriate for the pluralism we say we want in the discipline, and which quite frankly -- whether intentional or not -- come across as punitive to those who have thoughtful objections to the implementation standards and process.

[Tim Buthe HfP/Technical Univ of Munich & Duke University]

Thank you Sheena, Kurt, and Guest, for your posts.

On behalf of the Steering Committee, let me first of all express our appreciation for these posts and encourage more posts that endorse (or register disagreement) with previous posts. Even if posts contain no further information, they are very helpful in giving a sense of how widely shared specific concerns or interests are. Of course, a brief explanation as to why one (dis)agrees makes such posts even more helpful, but no such elaboration is required.

Second, speaking just for myself, could I ask you, Kurt, Guest, and Sheena, to elaborate on what the logistical burdens and costs are that you are most concerned about--and in which type of research (or for which kinds of transparency practices) you think they are most likely to arise?

I ask this because "qualitative research" encompasses a broad range of method (essentially all parts of anyone's research that is not just statistical or experimental). In the second stage of the QTD, we expect to form working groups to work through costs and benefits and practicalities of research transparency in a differentiated manner, so as to be able to take account of (many if not necessarily all) of the differences among qualitative methods. It would be very helpful for the purpose of deciding on the substantive foci of the working groups to have a clearer sense of what specific concerns are unique to a particular type of qualitative research and which concerns are widely shared.

Thanks!
Tim

[Jane Mansbridge Harvard Kennedy School]

My concern is that journal editors, who will be ones requiring that the submitting author do certain things (give reviewers access to data, compile a transparency appendix, etc.) will not necessarily be from the community of scholars doing the research. Occasionally even some of the reviewers may not be from that community. The revised APSA Ethics Guidelines (2012), as I read them, intended the new guidelines regarding transparency to be guidelines for the individual researcher. Members of that committee remember no discussion of the journals taking on the power to enforce “transparency,” as they later did. This deliberative process and subsequent ones will undoubtedly flesh out different models of what would be good practice in particular fields. So it is premature at best to begin imposing requirements now, as some journals have done. Indeed, it is not clear to me at this moment that the journals should ever impose specific requirements in qualitative research. If one or more reviewers thinks that certain steps should be taken toward greater transparency and the researcher disagrees, that should be an issue that the researcher discusses with the editor at that moment. In most cases the benefit of the doubt, I believe, should be given to a researcher who says that the proposed process would be too invasive of subjects’ privacy or too cumbersome and costly.

[Guest]

Jane's point that the standards should not be enforced by journal editors is well taken. The only DA-RT standard that journal editors could meaningfully enforce right now is access to data. And as various posts made clear, this is the most controversial as well as the less feasible standard for many qualitative research approaches. The other DA-RT standards regarding production and analytical transparency would only be enforceable by editors if they were to read the actual submission and be familiar with the subject itself. Clearly, this is neither likely nor, from the editors or discipline's perspective, desirable.
Given these problems would it not be possible to continue talking about formulating more careful DA-RT standards that could be used to guide the evaluation of reviewers. This would lead to a more consistent review process. Journals adopting DA-RT could explicitly state that the evaluation of those criteria falls exclusively to the reviewers and not the journal editors. You could think of this as the DA-RT subsidarity principle that transparency evaluations should be made at the most local or ontologically most proximate level.

[Michelle Jurkovich University of Massachusetts Boston]

Very insightful points, all. Tim, to respond to your request for elaboration on the specific “logistical burdens and costs are that you are most concerned about--and in which type of research (or for which kinds of transparency practices) you think they are most likely to arise burdens and costs for qualitative research,” I’d add the following (echoing concerns raised elsewhere):

1) TRANSCRIPTION COSTS: I’m currently working on an article that is built off of about 70 different interviews I conducted with anti-hunger activists. Obviously not all of them are cited in the article, but they all provided important information on context and background that shaped the argument. I tried to transcribe some myself, but this is not my gift (it took about 5.5 hours for me to transcribe 1 hr of interview material). Fortunately, UMass provided funding for me to have the remaining interviews transcribed (about 50 of them) at a cost of approximately $5000, as they were lengthy interviews. I used an affordable transcription firm that had been recommended by other qualitative scholars but was shocked at how expensive this is when you have long interview conversations to transcribe. Had my university not been able to fund these transcriptions, I likely would have been unable to submit this article to any DA-RT compliant journals. I am able to use my audio recordings and my own field notes when writing the article to ensure I’m quoting properly, but obviously audio files cannot be submitted (identifiers!) to a public repository. So I’d be up a creek if they were required. Had I needed transcriptionists to go through these interviews and remove all identifiers, the cost would have been far greater. Who will pay these costs for those at universities that cannot provide this funding?

2) NOT ALL ARCHIVES ARE PUBLIC: I’ve done archival research in archives where uploading images would be permitted (US and UK National Archives) though time consuming, but also archival work where I would NOT be able to upload images (i.e. individual NGO archives) or where getting permission to upload images would likely take at least several months to obtain (permission is granted case-by-case) and would depend on the documents to be uploaded. Sometimes I feel like people advocating for quick uploads (active cites) for archival docs are assuming all archives are somehow public and OK with whatever images you take being freely available. In my case, one archive in particular took about 16 months just to get permission to access, and they were quite clear that all images belong to the archive itself.

[Tim Buthe HfP/Technical Univ of Munich & Duke University]

Thank you, Michelle. This is very helpful; both points. On the first point, about transcribing interviews: While there are surely many circumstances (and types of research) where providing a complete, transcribed archive of interviews from a research project is (or would be) tremendously valuable for other, future research, an undertaking of that scale should surely be considered a substantial output in its own right (akin to something like an oral history archive), *not* something that can be just expected as a "required" appendix to a journal article (or even a book). Whichever working group will end up addressing such aspects of interview-based research during the 2nd stage of the QTD might want to not only make such an understanding explicit (if it is as widely shared as I believe) but also consider whether we, as a discipline, can at the same time develop better guidelines that would help identify the modest subset of interviews (or core elements of the interviews) that are sufficiently critical for the understanding or assessment or the research (or for subsequent follow-up research) that we would generally attach great value to making additional information from/about those interviews available.

[Michelle Jurkovich University of Massachusetts Boston]

Thanks, Tim.

I might add, though, that it's not clear to me how a journal editor would determine how many of those 70 interviews (from the example above) were really essential to provide in order to meet the objective of transparency. According to JETS "Transparency requires making visible both the empirical foundation and the logic of inquiry of research." If you're using interviews to answer a specific question (i.e. who was in the room when XYZ was negotiated? etc) then it's obvious which interviews would be relevant to submit, but if you're using interviews to understand an organizational culture, for instance, that will take far more than a handful, and they'll inform your research in complex ways. My hunch is it would be no easy task at all for journal editors to determine which interviews and which conversations would need to be provided and that it would likely be enough to be prohibitively costly to comply. Then, of course, there is the extreme job of annotating all these interviews so that someone who may know nothing about food policy would be able to make sense of them in their proper context. If I didn't do this (provide extensive background info on context with each interview) I would worry that a scholar who was not up to speed with this issue area would mine transcripts inappropriately, or that activist accounts could be used improperly.

The broader ethical problems are even more concerning to me. I had all these interviews transcribed for my own files, to aid in my future work. They were not produced for public viewing---in part because under the IRB protocol I used my respondents were promised anonymity with very specific guidelines over how I'd attribute the quotes I used. For projects that went through IRB before DART or JETS it's unlikely that scholars explicitly asked interviewees if the full transcripts from their visit could be made public, even if identifiers were removed. And that permission would be essential.

[Tim Buthe HfP/Technical Univ of Munich & Duke University]

Thank you, Michelle. These are all good and important points. (Let me add that, including a request for permission to create and share full transcripts publicly can very significantly change the nature of the conversation and for some kinds of research undermine the entire purpose of having or observing the interview/conversation, as Katherine Cramer pointed out in her contribution to the QMMR symposium; see http://ssrn.com/abstract=2652097 , esp. pp. 19f.)

Thanks again for your posts. This is exactly the kind of specific information that will help move the QTD forward.

[Danielle Lupton]

To build off of Michelle's second point, even those archives that are public, such as many Presidential Libraries, have quite restrictive rules regarding the dissemination of their material. At the Eisenhower Library this past summer, I was allowed to take photographs for myself, but it was made clear to me that such photos were only for personal use and that reproduction was not permitted without explicit consent. I also asked directly if I could share the photos with others and was given an ambiguous answer implying I should not do so unless the person was a direct co-author.

I find myself in favor of more transparency. My concern regarding my research (which is archival) is ensuring that the rules we adopt are feasible given the restrictions placed on us by other outlets (such as IRBs, archives, etc).

-Danielle Lupton
Colgate University

[Guest]

I agree with Danielle Lupton's excellent comments about the difficulty (if not impossibility) of sharing materials from archives. I have had similar experiences when conducting archival research.

I would like to build on her point by making the observation that this type of difficulty is not unique to qualitative research, even though it may be uniquely common to qualitative research. Specifically, quantitative studies in political science sometimes also rely on restricted use data that researchers are given access to (or have purchased access to) on the condition of not circulating it further. My understanding is that in the past, journals have been willing to make exceptions to their replication policies in such cases, though perhaps that is changing with the advent of DA-RT. (If it is, I think that would be a bad thing for many of the reasons already hashed out here and elsewhere.) It may be helpful for the deliberations here for us to think about what types of replication opt-out policies are (or should be) in place because of "limited use data restrictions", regardless of methodology or epistemology.

[Caroline Beer University of Vermont]

I agree with Kurt Weyland's post.

[Kristen Harkness University of St. Andrews]

I think Michelle's point that "if you're using interviews to understand an organizational culture, for instance, that will take far more than a handful [of interview transcripts], and they'll inform your research in complex ways" is very important and equally true for archival research. For example, I have drawn a set of inferences about intelligence reports reflecting real differences in the saliency of ethnicity within local communities, rather than differences in the reporting officers' individual biases, based on the shared training of those officers, their personal interactions, and their circulation between postings across east africa. Those claims, in turn, rest on hundreds of documents, drawn from multiple countries (including requisition reports for things as trivial as soap) that collectively show how intelligence officers moved from post to post and corresponded with each other. It would be possible to post or hyperlink all those documents. But as others have pointed out, that would be enormously costly, prohibited by some archives, and would replicate the work of archivists.

[Guest]

I agree with the posts by Kurt Weyland and Sheena Greitens at the top of this thread.

[Alyssa Grahame UMass Amherst]

I tend to agree with other posts that point out that these requirements would impose an undue burden on qualitative researchers. As a graduate student, I have had the great fortune to receive external funding to support field research, which produced original data. I do not have funding for interview transcription or field note transcription. As I write, I use my notebooks, transcribe and code interviews selectively, and listen to other interviews as needed. To produce full transcriptions of all data on my own would require at least another year on my PhD, which is simply not feasible for a number of reasons.

[Mneesha Gellman Emerson College]

I agree with Kurt Weyland's post as well, for many of the reasons already specified by other people. It imposes an undue burden on researchers and can act as a disincentive to follow this kind of research path, particularly for those of us at non-R1 schools with limited research funds to pay for transcriptions and the other layers of data organizing necessary to take private data and make it public.

[Guest]

I would like to add my agreement to Michelle Jurkovich's post above and emphasize the point she makes about differences in archives in different countries.

[Michael Coppedge]

I agree with Kurt Weyland. This discussion presumes an idealized notion of academic deliberation in which many scholars are attentive to everyone's work in their field and are actively challenging and questioning every argument and therefore always needing better documentation. In reality, the median number of citations of academic journal articles is zero. Most published work is ignored, and much of what does get attention is not challenged very seriously. Who has time for that? Requiring greater transparency would be a waste of time in most cases. A certain amount of checking is done by referees and editors, but even they are already overburdened and few or none would have the patience to pore over exhaustive documentation of archival materials, interview transcripts, or observation notes in order to replicate an author's conclusions. In most cases, we have to rely on a certain degree of trust in one another's professional honesty and diligence. The alternative involves very high costs for very little benefit. There are exceptions: seminal or controversial pieces that provoke debates among many people. These debates would benefit from better documentation and transparency. But such documentation can wait until we know that a debate has arisen, when it can be published in follow-up articles or research notes or, if the author prefers, an online archive (confidentiality permitting). This is basically what happens now. I am not unhappy with the status quo. I support as much transparency as possible in quantitative research, including sharing datasets and scripts with referees prior to publication. But for qualitative research, I see this effort as an impractical solution in search of an infrequent problem. Michael Coppedge, University of Notre Dame (waiting for approval of registration)

[Tim Buthe HfP/Technical Univ of Munich & Duke University]

Thank you for your post, Michael. I read Kurt Weyland's post as registering opposition to JETS-type requirements imposed by journal editors as prerequisites for publication. Your post appears to be a much more sweeping rejection of the entire QTD project.

That's of course fine--and indeed it's important to know if many colleagues feel this way--but I've been surprised for years, whenever I teach my PhD seminar on "Research Design and Qualitative [=all non-statistical] Methods" (for syllabi see http://people.duke.edu/~buthe/downloads/teaching/syllabus_ps732_s2015.pdf and http://people.duke.edu/~buthe/downloads/teaching/syllabus_ps330_s12_public.pdf) and when I talk with graduate students at Duke and elsewhere, that current PhD students--those most junior, most resource-poor scholars on whom the burden of greater research transparency would disproportionately fall--seem generally little concerned with the costs in terms of time or material resources. Instead, they are quite enthusiastic about the prospect of stronger transparency norms and practices that would result in research that would allow them to better understand what the scholar-authors did and how they did it. They are eager to learn from it--and to be able to better assess the papers, articles, and books they read (a few have also noted that they would cite more of what they read if they felt they had a better sense of how the results were arrived at). And they are generally very willing to embrace technological solutions (and sometimes to contribute to them) that would reduced the costs of achieving greater transparency.

So rather than categorically asserting that the light gained from deliberating over transparency isn't worth the candle, might it not be more useful to be specific about the costs, risks and trade-offs in particular (identified) transparency practices about which we (often surely very sensibly) have serious concerns?

[Guest]

Perception and Misperception in DA-RT

As a junior scholar, I complete agree with Tim. In addition, as a scholar who views methods (qualitative, quantitative, formal) as a toolbox to answer questions of interests, I am surprised to see so much agreement with Kurt.

It seems to me that perhaps the reason why most here believe that transparency requirements place an undue burden on qualitative researchers vis-a-vis quantitative researchers rests on a perception that all quantitative researchers do is:

1. Google for dataset
2. Run analysis
3. Win.

To satisfy the transparency requirements, quantitative researchers simply have to:

1. Open favorite text editor
2. Write R or Stata code
3. Save and upload to Dataverse or personal website.
4. Win.

If we compare this to the typical perception of qualitative researchers who have to:

1. Think hard about survey/interview questions
2. Write IRB protocols (ugh!)
3. Visit archives, region of inquiry, interview participants ($$$)
4. Transcribe recordings, take pictures of historical material ($$$)
5. Write up analysis, which is a helluva lot longer than the results section of quantitative papers
6. Face bias from (top) journal editors and peer reviewers

To satisfy the transparency requirements, qualitative researchers have to:

1. Obtain informed consent (ugh!)
2. Inform subjects that any information provided may be accessible to the general public, which may change their responses, or in some instances, not respond at all
3. Spend money to engage transcription services, which some of us at lesser institutions (like myself) do not have
4. Write letters to request permission from relevant organization
5. Spend time writing "meaty footnotes", during which our quantitative counterparts' manuscript is already under review
6. Face bias from (top) journal editors and peer reviewers

This difference in perception, which hearkens back to the latter half of the 20th century, gave rise to the Perestroika movement in the field, and continues to divide departments today, is perhaps the central reason why we are engaging in this debate. This is akin to saying I (a qualitative scholar) am being penalized for crafting my own hammer while all you (quant researchers) did was buy yours at Home Depot. What's worse: while you can simply show your Home Depot receipt to the editor, I have to make sure to keep every receipt from each component I bought to construct my hammer. Too. Many. Receipts.

In one sense, this discussion is similar to what I see in psychology and medicine: Due to issues of privacy and anonymity (HIPAA! IRB!), we cannot share our data. It's not because we don't want to (we love transparency and are ethical!), but because we care about our subjects. I don't have to expound on what this attitude has done to their reputation and credibility. Psychological Science anyone?

At a more fundamental level, I see the vehement disagreement over DA-RT as stemming from the belief that it exacerbates an existing biased publication process against qualitative scholars. What is more, it uses "transparency" to mask this bias: "Look, I (top journal editor) am not biased. I just want you to show me that you did the work correctly and to the best of your ability, and if any of our colleagues would like to replicate or reproduce your work, they can easily do so instead of waiting for an email response that almost never comes. So, erm, reject because you didn't provide enough data."

It'd be disingenuous to say that these perceptions are unfounded; however, I do think that there is some misperception at work:

1. Restricted use policies are present in both archives and quantitative datasets. See Nathan Jensen's travails on getting an excellent article published because the dataset he used was classified.

2. As some of my colleagues have pointed out, irrespective of method, conceptualization and measurement is paramount. In any paper, regardless of method, we must adequately defend how we conceptualize and operationalize variables of interest. This should be a point of non-contention.

3. The "google for dataset" process is less easy that it seems. There is also an increasing trend towards the generation of original datasets, which requires extensive documentation on the part of researchers.

4. Our quantitative colleagues are not "out to get us". Rather, they recognize the "garden of forking paths" problem that quantitative researchers often take, and which are detrimental to the field.

One point of pride I have as a political science scholar is that I can declare that we are the only discipline who makes datasets and code publicly available. We, political scientists, provide public goods. We, political scientists, allow our colleagues and the three members of the public who care about our work to trust and verify our work. This is not the case in our fellow social and behavioral scientific fields, where clientelistic relationships prevail: "I (full professor) have a ton of data from my taxpayer-funded R01 grants from NIH. Only students whom I deem able can analyze my data, and no one else. If you have concerns about my data, I have five words for you: Trust me. I'm a professor."

If, as Gary remarked in a different thread, that because of DA-RT, qualitative researchers will be held to a higher standard, we should welcome that. If we are concerned about bias in journals, or the increased time-to-publication effect, then Rank and Tenure committees should adjust their expectations accordingly. This is something departments can do if things are really that bad. More importantly, if there is indeed such a bias, then those who sway over the field (clearly, not me, being at a second-rate institution and all) need to step up and do something. Before that, however, perhaps it's time we took another look at the methods in our top publications?

I'll be the first to admit that my research process is less organized than I'd like. I have multiple articles/datasets/code in different folders across different machines. I have books on different desks and shelves in different locations. I have notes and interview transcripts in my office and at home. I have audio recordings in different folders across different machines. Hmm, maybe this is why I'm not at an R1...why won't you sync, Google Drive?! Sync!

I recognize that DA-RT will force me to develop new habits (which is painful) but it is like starting an exercise regimen. Once it becomes part of your life/workflow, I am better off for it. I do love Netflix though, and my couch is soooo comfy...

A final point: one thing missing in this discussion is the recognition that researchers who use a formal toolbox and analytic narratives/case studies would probably face similar considerations as qualitative researchers.

[Robert Fishman Carlos III University]

Sorry not to have been able to join this exchange earlier. My initial instincts are similar to those of Kurt Weyland and I strongly agree with the elaboration of that view expressed by Michael Coppedge in his May 20 post. All scholars should be prepared to defend their work when it is subject to criticism or questioning but rigid a priori and formalistic requirements do not strike me as the best way to attend to this need. Instead, it is serious scholarly interchange on research-based arguments which ought to be encouraged in the appropriate venues of such exchange -- in print, at professional meetings, etc. In the context of such exchanges reasonable questions ought to be raised and should be adequately answered.

[Alan Jacobs University of British Columbia]

Thanks to all who have been contributing to this very interesting exchange.

It is clear that there is considerable discomfort with the idea of uniform, rigid editorial transparency rules being applied to the assessment of qualitative journal submissions. Some, like Michael Coppedge (above), have also expressed the view that qualitative work is already sufficiently transparent, and that there is no problem to be solved.

I am wondering about a couple of things, though. One is whether we need to think about transparency as a solution to "a problem." Does the case for transparency hinge on evidence of fraudulent behavior or routine (if perhaps inadvertent) misreporting? Could we not think of transparency as providing a benefit -- such as enhancing the interpretability of research results, helping scholars understand how findings or interpretations were arrived at and thus what the implications or limitations of those findings/interpretations might be? Do we think that qualitative research generally does as much as can reasonably be done to provide such a benefit?

Second, thinking ahead to later stages of the QTD, I am wondering what people think of approaches to advancing or assessing the transparency of qualitative research that do not rely on rigid and uniform editorial rules. One approach might be norm-based: for qualitative researchers to identify principles, valuable practices, or evaluative questions that researchers might use in designing and writing up their own research and that might inform reviewers' and editors' assessments of individual pieces of research. These norms might to a large degree articulate current practice, but help draw attention to valued practices and perhaps uncover ways in which those practices could be extended, refined, or better supported (e.g., through institutional infrastructure or editorial policies -- e.g., longer word counts). Rather than uniform, these norms might also be articulated in differentiated ways for different logics of qualitative inquiry or evidence. And they might apply to transparency in a very broad sense -- including perhaps explicitness about positionality or about the iterative process by which the researcher moved between theory and observation. The QTD itself might contribute to such a normative approach via its Working Group deliberations and Community Transparency Statements.

Another possibility, suggested by the conversation above, would be to generate more robust forms of post-publication assessment. In principle, open scholarly debate would seem a powerful way of advancing many of the basic intellectual goals that transparency advocates seek to pursue. In practice, however, as others have observed, there are currently few outlets for critical commentary on or debate of individual pieces of published work, aside from the publication of a separate, original piece of research. Even in the latter, engagement with the empirical bases of previous work is usually quite thin. The creation and broad use of fora for engagement with published work -- perhaps online discussion boards on publishers' websites? -- might yield greater explicitness/openness/transparency quite organically as colleagues raise questions of evidence, interpretation, inference, and ethics and authors seek to make the pieces they publish more carefully grounded from the outset in anticipation of ex post scrutiny.

These are just a couple of possibilities. What do you think might be useful ways of encouraging or promoting greater transparency or explicitness, however we might define those concepts?