Qualitative Transparency Deliberations

[Erik Bleich Middlebury College]

In keeping with Deborah Yashar’s encouragement to undertake reflections on the meaning of transparency for different qualitative approaches, what are the pros and cons of transparency for those working with interview-based research?

In my own work, I have found that following transparency guidelines helps me think through how to decide what types of actors to interview, what types of questions to ask, and how to provide summaries of those interviews that can be shared with other researchers who were not in the room on the day.

The primary downside has been the time it takes to provide the summaries and to make sure permissions and protections are in place, especially when dealing with a foreign language and translations.

My interviews are mostly with elites in established democracies. There may be different pros and cons to transparency across elite/non-elite interview and democratic/non-democratic regime divides.

What are others’ experiences around transparency with respect to interview-based research? What are the primary benefits and drawbacks for the kind of work you do?

To see what other contributors have said so far about interview-based research, have a look here:
-Re:[From Steering Comm.] Inviting input on specific transparency practices
-Very Undesirable Unintended Consequences
-Transparency, openness, & ethnography in democratic contexts
-Harms in study of identity, and forgotten agency
-Re: Right to be Forgotten?
-Against “requirements”

[Derek Beach University of Aarhus]

From someone who has done extensive elite interviews amongst high-level civil servants in both EU institutions and EU member states on highly sensitive topics (treaty reform negotiations), I can confidently say that if my respondents knew I would be forced to archive a transcript - no matter how I tried to mask their identity - they would never agree to talk to me. I would therefore either have to work with poorer sources to live up to DA-RT norms, or I would have to choose non-DA-RT outlets for publication.

The standard should be that the scholar provides enough information about the set of interviewees, thereby enabling other researchers investigating the same topic to be able to find similar things by interviewing the same or similar people.

If transcripts have to be made available in some form to avoid researchers making things up (is this really a problem?), we should instead of a DA-RT archiving engage in what journalists do - a confidential review process with an editor who has access to the full confidential material. In a good journalistic practice, the journalist has to convince the editor that the evidence of an event is not made up or is biased, but instead comes from multiple independent sources. We could adopt similar practices, where the editor of the journal has to be provided with access to the full empirical record that documents key claims in an article/book. (I would be hesitant to give access to this material to reviewers, given that these are often competitors in the field...)

The advantage of introducing a confidential editorial review process would be that we can protect the anonymity of our respondents, and we would have better empirical research because we as researchers would be forced to really make the case towards a critical editor that our evidence actually enables the claims we think that it does.

[Erik Bleich Middlebury College]

Thanks for your reply, Derek. I agree that if interviewees felt that transcripts *had* to be posted, they would be much less likely to talk. But I don't read DA-RT that way (though I realize there is room to read it that way, if an editor who has signed JETS insists on doing so).

In my work, I have said at the end of the interview something like, "It is becoming increasingly common for scholars to archive summaries or transcripts of interviews so that future scholars can learn from our conversation today. Do I have your permission to put a summary/transcript of our conversation in such an archive?" Many have said "yes"; some have said "no"; and most have said "yes, but please send me the summary/transcript beforehand so I can check it for accuracy."

Given your experience, do you think something like that could alleviate the potential downsides we are all worried about?

For me, it has worked, but it also means a fair amount of time to go back and forth with the ones who I didn't get a clear "yes" from, especially given translation issues. I do worry that sending these folks back a more formal form asking for permission--which I am working on now--may discourage future interviews with them (or with their contacts, if word gets around).

[Guest]

I have conducted more than one hundred elite and non-elite interviews in developing countries on slightly sensitive political issues. To date, I have only conducted interviews with the guarantee of anonymity. I know that for some of my interviewees, it was important that they remain anonymous, but in retrospect, I believe that some of them would have been willing to go on the record and have allowed summaries and transcripts to be archived. Some of them may have even appreciated that because it would have given them some public recognition. There is something to be said for trying to create interview summaries or transcripts when interviewees are willing and there is no foreseeable risk to them, and I'm open to moving in this direction in my future research when it is feasible for me to do so.

However, I wish I had more evidence to guide my decisions about how to conduct interviews, rather than having to rely on my gut instincts and the relatively small literature on interview methods in the social sciences. Many on this board have written that they would experience profound costs if they pursue on-the-record interviews or start to make interview transcripts available -- both in terms of their time and in terms of their ability to gather accurate information. I am sure that that is the case for certain research topics, but I'm not sure that it is the case for my research topics. It might be, but it might not be. Something helpful that these deliberations could inspire would be more research on field methods to help us assess the counterfactual of what would be lost and what would be gained if some of us changed our interview techniques in favor of greater "data access and research transparency."

[Michelle Jurkovich University of Massachusetts Boston]

Of course knowing that a full transcript would be made public would alter what an interviewee would say. Interviews are inherently social exercises where it matters very much who is doing the interviewing and the level of trust a respondent has in the interviewer in telling them about their work, their beliefs, etc. This trust is generated at times through repeat interactions, or through a referral from a trusted colleague or friend, etc, but it’s an important part of the method. An interviewee isn’t answering questions to the public, they’re answering questions to the human sitting across from them, often in a room where the door is closed and the environment is set up in a way (very intentionally) to create a feeling a privacy. The terms of disclosure are set from the beginning per IRB, but I’m positive if I went into interviews with activists and prefaced our meeting by saying everything we said could be made available to whomever wanted to read the transcript (which to these individuals would also make them think potential donors or other activist groups might also get their hands on the transcript, even though that'd be unlikely) that the tone and substance of the meeting would shift very quickly to, “you can read everything you need to know on our professional website” and similar stock answers.

Moreover, an interviewee, if speaking to the public, would likely word their responses very differently than they would when speaking to someone they know already has a great deal of important context and background information going into the conversation. I would be doing my interviewee a disservice if I released their raw transcripts without filling in all these blanks as I would very much worry how these interviews could be misinterpreted by individuals who do not have the necessary background to understand their meaning. And this process of annotation/providing notes on context and background about what was discussed, etc would be onerous.

[Erik Bleich Middlebury College]

Just thinking out loud here in response to the previous two posts. Is it possible to develop a sense of "best practices" that would allow each interviewer the room to judge when, how, and whom to ask about archiving? IRBs already dictate some of the terms of the interaction for many types of interviews. But there are some interviewees who want to be archived. Not just average people who want their voices heard, but in my case, well-placed judges who feel that the public has for too long been deprived of information about what happens in French courts.

In my own interviews, I am careful not to raise archiving until the end, so as not to affect the nature of the interview. I am also careful to stress that the archive is not going to be accessible to all, but just to bona fide researchers. I am equally careful to respect what the interviewee says s/he wants, and to attend to hesitations in their reactions by not simply bulldozing my way to a yes.

The summaries I am sending them back for their approval reflect a bit less than what my notes reflect. If I give the "metadata" of my reactions to their statements, they might not approve archiving at all (and it might shut down future conversations). So what's in the archive will not be perfect. But that is true of all written archives as well, where one has to be knowledgeable enough to read between the lines to interpret the information appropriately and not simply to take it at face value.

In short, is there a workable standard we can develop that avoids the most egregious pitfalls of the project while still increasing transparency? And can we do this while putting more trust and faith in individual researchers to make the call about when, whom, and how to ask about archiving?

[Michelle Jurkovich University of Massachusetts Boston]

IRBs may vary in their consent protocols but in my case, at least, interviewees were required to receive an information sheet before the interview begins as an important part of the consent process. One of the things that my IRB requires is that information about how data will be stored, who will be able to view transcripts, and how quotes will be attributed all must be discussed prior to obtaining a verbal consent to begin the interview. Time is set aside in the beginning too to allow the interviewee to express any concerns or questions they have about who might have access to the things they say. Now, of course these things can be raised again at the end of the interview, but they can't be avoided at the beginning----which is why I noted above that informing respondents that if I wanted to publish in certain journals these transcripts could be made public on online databases (made available to anybody with a paid subscription to the journal, or with access to a library that has this subscription, presumably? which could be, well, anybody) would certainly alter the tone and substance of our conversation. All this to say, my hunch is that my institution doesn't run the only IRB that requires these things to be negotiated as part of the consent process (which obviously happens before the interview itself).

In reflecting on Erik's question ("In short, is there a workable standard we can develop that avoids the most egregious pitfalls of the project while still increasing transparency?) I'd echo the question asked by several others on this board already: for what problem is DA-RT meant to be the solution, particularly in reference to the use of the interview method? I doubt we can have effective conversations about generating solutions until we have an agreement on what the problem is in the first place.

[Guest]

To second Michelle's comment: most IRBs I've encountered require that you ask about recording, transcribing, confidentiality, etc at the beginning of an interview, as part of the process of obtaining informed consent. I know for a fact that my current university IRB would not consider it ethically permissible to wait until the end of an interview to ask about posting a summary or transcript (in their view, it borders on deception, and violates the requirement for fully-informed consent). Different university IRBs are not necessarily consistent with each other, but my sense is that that is the general standard as well.

[Mala Htun Univ of New Mexico]

What Michelle said--"IRBs may vary in their consent protocols but in my case, at least, interviewees were required to receive an information sheet before the interview begins as an important part of the consent process. One of the things that my IRB requires is that information about how data will be stored, who will be able to view transcripts, and how quotes will be attributed all must be discussed prior to obtaining a verbal consent to begin the interview"--is the case with my IRB as well.

I like the idea of archiving some interview transcripts for the public good--if the project's nature is amendable to this (meaning absent of situations where subjects would be endangered, feel intimidated, and with conditions enabling them freely to consent to it). But the goal here is increasing the knowledge base available to future scholars, not achieve a meta standard of "transparency."

What Erik said--"The summaries I am sending them back for their approval reflect a bit less than what my notes reflect. If I give the "metadata" of my reactions to their statements, they might not approve archiving at all"--explains precisely why interview archiving is not the same as transparency, and could be at odds with it.

Inquisitive scholars who looked just at the redacted or anonymized transcripts would not be able to come to the same conclusions as Erik, b/c they would be taken out of context, and missing the surrounding impressions and intuitions--which are just as crucial as the interview text--that enabled him to arrive at his findings. As Cramer put it in the QMMR symposium from last summer, the data consist in "being there," and thus remain "in her head," not just in the recorder or on the page.

in short--making transcripts public is good, but it's for the common good of preserving and expanding access to knowledge, not to conform to a standard of transparency. i agree with discussion on other threads that transparency means "explicitness."

[Lee Ann Fujii University of Toronto]

Mala makes an excellent point that "archiving is not the same as transparency, and could be at odds with it."

I have just finished a book manuscript on what I call "relational interviewing." From the beginning, I wanted to include some interview snippets in an appendix so that readers could see for themselves how "real" interviews unfold. But when I actually read the transcripts (to decide whether to keep them), I realized how reading a transcript of an interview (including my own) was so completely different than being there in real time. I realized how much transcripts leave out, all the unspoken forms of communication, to be sure, but also the history of a given relationship. When I interviewed the same person a 4th or 5th time, that interview was embedded in a relationship that the participant and I had built over months and sometimes years. Needless to say, that history does not "appear" in the transcript. Of course, we could (and indeed *should*) fill in the context when we analyze, write up, and present our work. But since very little of the meta-data and interactional dynamics get into the transcript, the transcript ends up being a very unfaithful record of the exchange. And as Portelli teaches us, transcripts are themselves a mediated version of the original interaction. Where we decide to place commas, ellipses, paragraph breaks, etc.--those are all acts of interpretation, he argues. So transcripts are not the kind of "raw" data that DA-RT proponents seem to think they are.

I also wanted to second what Michelle said about the effect of adding yet one more item to the IRB consent protocol that basically calls into question our promise to keep identities confidential. How can I really say to interviewees that I will protect their identities while at the same time "informing" them that I might have to post a written transcript of our exchange to public repository? I believe that kind of "promise" is called double-speak.

[Brian Taylor Syracuse University]

I have not had time to read through all of the interesting and thoughtful comments in the online deliberations, but given that time is running out for contributions, I wanted to add some personal reflections based on my own research. Some of these points have been made by others, whereas others I have not seen yet, at least in the threads I had time to read.

I do research in Russia, and much of my research involves state coercive agencies – police, secret police, prosecutors, military, and so on. Some of my respondents are current of former employees of these agencies, while others may be journalists who write on these topics, NGOs who interact with these agencies, scholars who study these issues, political figures who also interact with these bodies, etc. When I first started doing interviews in Russia in the early 1990s, I was advised not to try to record interviews, for the purposes of encouraging candor. I have kept to that practice since then. What that means in reality is that my notes are necessarily partial, sometimes in English and sometimes in Russian. I particularly use Russian to write an exact phrase that I think I might like to quote verbatim. I go back over the written notes immediately afterwards and fill in gaps, and then as soon as possible afterwards type up my notes.
When writing articles and books based on this material, it is often the case that inferences drawn from these interviews – about standard practices in an agency, its organizational culture, and so on – are based on not only multiple interviews but newspaper articles, NGO reports, other scholarly work (in English and in Russian), government reports, etc.

What does this have to do with DA-RT? Several things.

+ As has been pointed out by others, securing interviews, in particular candid ones, would be made considerably more difficult if I had to get permission to post a transcript of our discussion online. Even if anonymous, it would be difficult to reassure someone that identifying information would not become public – especially since it might be hard for me to decide what information is identifying, or might be identifying at some point in the future.

+ This point is related to a different part of the forum, about working in authoritarian contexts -- talking to an American academic in itself can be seen as a risky endeavor in contemporary Russia. More than one person mentioned to me last summer that even talking to me might put someone under scrutiny for being a “foreign agent” (a few years ago Russia passed a law on NGOs and “foreign agents”). I feel pretty confident that I would have more difficulty securing interviews if I had to request permission to make public a full or partial transcript.

+ If some editors start expecting interview transcripts in the name of data access and research transparency, how long before some IRBs start requiring that scholars using interview methods secure consent for publishing transcripts? That is, even if I personally decide I would not publish transcripts from interviews because of the concerns above, it strikes me as plausible that at some point in the future cautious IRB personnel might decide that permission for publicy-available transcripts are an important part of informed consent, if certain versions of data access become more broadly accepted. I realize that this point is hypothetical, and may be overblown, but it seems worth thinking about this issue in advance.

+ My narrative above about my field practices suggests that the transcripts I produce might be of limited utility to someone not already steeped in the issues with which I am dealing. Things that my respondents and I take for granted might not be clear to other outside observers. Say, for example, I hear a story about a particular law enforcement or political figure in a certain locale. I may have to use multiple interviews and other sources – newspaper accounts, for example – to make sense of the story, and verify it. Any scholar who works closely on the issue might be able to replicate some parts of the story also. What would an overworked journal editor, who is not an expert on the country or topic, make of a fragment of one interview on that topic? Perhaps it would be incumbent upon me to provide a fuller narrative of how I interpreted the story and tried to verify the salient points, in the interests of analytical transparency. In principle, that sounds reasonable to me. In practice, it sounds extremely daunting to do this for every source used in an article or book, and every footnote in which these materials are cited.

+ Footnotes remain a vital piece of data and analytical transparency. It has been noted elsewhere in the discussion that the community of scholars working on a topic are the ones who ultimately decide on the value of particular published research products, and are in the best position to replicate the type of interview-based research under discussion here. It is not that some future scholar will necessarily go interview the same cop from some town in Siberia. Rather, they can look at the body of evidence provided in the footnotes as a whole – newspaper articles, NGO and government reports, interviews, and so on – to judge the value of the work. They might even be conducting interviews with similar types of respondents, and be in a position to say, in their own work, how (in)accurate the findings of other scholars appear to them.

To conclude, I would add my voice to those who are concerned about what the implementation of DA-RT standards by political science journal editors might mean for researchers using interview methods, particularly in challenging circumstances (because of the countries that they work in, or because of the topics they study). I think the proposed cure might be worse than the disease – and I’m not even sure what the disease is. If the disease is that it’s difficult to replicate certain types of political science research, that should be obvious to everyone in the field. All of us can think of aspects of certain works that would be very difficult if not impossible to replicate – certain ethnographies, certain type of interviews, and so on – but they still contribute to our body of knowledge.

I particularly fear what new and uncertain research standards mean for young scholars making their first foray in the field in a challenging environment, gathering data that they might be using for the next decade. Given that it will be hard for them to know at this point what a future editor might ask of them in the name of DA-RT, I hope that they are not discouraged from doing certain types of work, and asking certain types of questions.

[Julie George Queens College and The Graduate Center, CUNY]

I'm a chair of an IRB. Right now our IRB standard procedures require that researchers disclose with whom, and in what form, data from research will be shared. The IRB office also requires disclosure about sharing possibilities on the consent document, which must be shared prior to the interview.

I conduct interviews in former Soviet space and offer confidentiality to interview subjects when they request it. This protection is in the consent document. If requested by journal editors, I can supply my IRB approval letter and consent documents to indicate my unwillingness to defy federal regulations in order to fulfill their mandate.

I, like others, am frustrated by the vagueness and unevenness of DA-RT and journal editors in expressing how the procedure would work in particular cases.

[Guest]

In response to the last post I ask, is it your opinion that the policies on these matters were clearer before DA-RT? I think not. Few journals had explicit policies and many decisions were being made on a case-by-case basis. I read DA-RT as a way to encourage journals to be clear about what their policies actually are.