Qualitative Transparency Deliberations

[Alan Jacobs University of British Columbia]

Over in a thread begun by Gary Marks, an interesting discussion has started about the nature of qualitative data access. To briefly frame the issue: There seem to be a number of different understandings of how qualitative scholars might make the evidence underlying their claims more accessible. One version that has received considerable attention would involve authors providing/embedding links (perhaps via active citation) to, say, the 2-3 crucial pieces of evidence supporting a claim. These links might take the reader to somewhat extended excerpts from primary documents or interviews. This form of data access is much less onerous than, e.g., providing full transcriptions of all interviews or scanning in all of one's field notes; and it would provide more information about the basis of authors' claims than readers currently get. This is akin to what quantitative scholars are asked to do in providing the dataset as analyzed in producing reported results. At the same time, some worry that this approach only reveals the "tip of the iceberg" in a long process of inference, or could introduce perverse incentives.

What do you think? Does providing key pieces of evidence in more extensive form help readers better understand and evaluate the empirical basis of findings? Is it enough? Does it heighten incentives to cherry-pick? Are there other not-too-onerous ways in which authors can usefully reveal more about the evidentiary underpinnings of their claims?

[Karen Alter Northwestern University]

I guess that I just don't see the problem this solution remedies--although the same is true for most of the DA-RT proposal.

I have no problem embedding links to publicly available sources-- that is easy to do, and I am happy to do so. It is the sources that are not publicly available that are the challenge. When a scholar is really interested, they write me and I share the documents. But I can't post these documents-- they are not mine to make publicly accessible. The documents I am discussing are collected during field work. Some of the documents should be public, such as reports and decisions produced by international institutions. Some are interim drafts that are not mean to be public. In either case, some higher official made a decision to not make the documents public. I can lose future access, or get my source in trouble, if I post the documents.

I will share the documents with legitimate scholars-- those who explain that their research project. And I keep my archives, which I also share with future scholars. This seems to work fine.

[Sherrill Stroschein University College London]

I agree with everything that Karen (if I may!) said above.
I presented a paper on this at a conference recently and was asked what we should do instead, to solve the problem of transparency. And I had to respond that I simply didn't see what the problem was. I have had people email me and request quite extensive information which I have then provided, in return for those parties agreeing to use the material wisely. What this means depends on the type of material. There really is no one-size-fits-all means to deal with this. Also, as confidential interviews on sensitive subjects require a commitment I have made to the source, I would not share this material but would rather summarize it carefully on request. That's as far as I can go without harming the source.
I can't get over feeling like a huge problem has been made out of a few instances of deception or individuals refusing to share their material. But such cases are also best dealt with individually. Taking the risk of harm that has been now extensively outlined simply makes no sense. This all feels like a manufactured problem.

[Guest]

How do journalists handle the materials collected and their sources? They don't put them out all for the public to view, but editors play the role of vetting the sources. Academics can do the same. Make the editor-author relations a privileged one, so the author can reveal the sources and qualitative data that has gone into the articles to the editor, and the editor only. In the meantime, the author should protect his or her sources from the general readership, especially if he or she has promised anonymity.

[Kristen Harkness University of St. Andrews]

I would like to add some practical concerns over developing widespread expectations of embedded links to key pieces of evidence, or even their provision in appendices. Others have talked about limitations in the context of interviews, private documents, or draft reports acquired from individuals or organizations during field work. There seems to be an assumption, however, that embedded links would be non-problematic in the context of archival work, which I would like to challenge. To preface, I have conducted archival research in the United States, Europe, and Africa both to build quantitative data sets and to conduct qualitative case studies.

I see two primary obstacles to the ideal of embedded links or other public provision of primary sources. First, some countries have copyright laws that specifically forbid publicly releasing their documents without consent, and some of them are unlikely to give consent. This has been an obstacle for one of my coauthors and I in obtaining and digitally scanning old British colonial maps for the purpose of constructing historical covariates for conflict research. We had to agree not to give public access to the digitized maps according to crown copyright law. Those same laws apply to the UK documents I have relied on in my case study work. Even if an agreement could be reached with the UK national archives for embedded links, perhaps to their website, there would be large transaction costs for each published article as the relevant documents would need to be digitized by the archive to their standards and permissions worked out and signed.

Second, some of the African archives I have worked in do not allow digital photography and do not have the capacity themselves to digitize documents. Obtaining photocopies is usually possible, which could be digitized later, although there is often a bit of an expensive racket around doing so. Those costs cannot always be well-anticipated up front. As a graduate student, that became a problem since I did not have extensive and flexible grant money to cover unexpected costs. I thus relied on extensive notes combined with meticulous citation for most documents while getting copies of what seemed like critical documents, which was the best practice possible in the circumstances. Of course, the later writing process always reveals the relevance of some documents that I didn't consider critical or all that useful upon discovery. With greater funding possibilities at this stage in my career, and better knowledge of how the archives I frequent work, I can avoid some of these issues. But they will still be relevant for PhD candidates. I think we also have to be very conscientious and careful about how we interact with underfunded archives in many countries and whether we publicly release their documents when they don't have the capacity to do so themselves.

[Sherrill Stroschein University College London]

In trying to post a more productive reply on this, I see that others have beaten me to it. So I would like to emphasize what has been said above...

1. Journalistic standards have been reached after a great deal of the field's own professional development. They involve getting at least 2 sources for a claim, and anonymity for those sources. These standards seem reasonable for research.

2. In a fit of interest in transparecy, I tried once to put together a project that would make public several thousand of digital photos that I took of newspapers in my region. I quickly learned that while libraries might be places of transparency, the newspapers still own the copyright such that I could only make the materials public with their permission. Thus, there are some potential legal violations if researchers are pressed in this direction. I could only get one of 8 papers to agree, as they are involved in their own digitizing projects, as mentioned above.

3. (Haven't seen this mentioned, apologies if missed.) Fieldwork is something that happens every minute that a person is in the field, if they are doing it right. Clearly material cannot be submitted for every minute, and interviews and even field notes are only part of what is being absorbed. What is the aim of DA-RT in this respect? Is it to "prove" that we were really there (can submit passport stamp copies)? Is it to try to retrace thinking through field notes? Could we come up with a minimum standard that would be acceptable and just muddle through with that?

4. As a journal editor I would be most grateful for some language that I could pick up and use in response to reviewers who might insist on some of the DA-RT requirements. As a reviewer for journals, I am considering whether it might be appropirate to include a section of my review for DA-RT compliant journals outlining why the piece submitted should not be subject to DA-RT if some of the problems from these discussions are clear. Maybe we could organize this if we can't get changes through?

5. Do we have to wait until a source sues a journal for the objections to be listened to? I am so glad for this discussion but in discussions with DA-RT evangelists I am not getting anywhere myself. Just patronizing language that it will all be ok because journal editors will figure it out. As a journal editor myself, I know we are not superhuman and thus am not encourage.

[Sarah Parkinson Johns Hopkins University]

With regards to Sherrill Stroschein's point #4 above:

"4. As a journal editor I would be most grateful for some language that I could pick up and use in response to reviewers who might insist on some of the DA-RT requirements. As a reviewer for journals, I am considering whether it might be appropirate to include a section of my review for DA-RT compliant journals outlining why the piece submitted should not be subject to DA-RT if some of the problems from these discussions are clear. Maybe we could organize this if we can't get changes through?"

I wonder what people think of ASR's reviewer guidelines for qualitative and ethnographic work, posted here:https://docs.google.com/document/d/1GYyw-S3K5TRYzPXsXJ2jPcgTO0G39nSo9e9fdwiOvuM/pub

[Guest]

That is very useful - thank you! I think it could also come in handy for concrete guidelines to come out of this discussion.