Page 1 of 1

Pursuing transparency in qualitative research

Posted: Tue Nov 08, 2016 6:58 pm
by ailitripp
It is not possible to replicate qualitative work in the same way as quantitative work, especially given how idiosyncratic fieldwork can be. No one is realistically going to replicate interviews that are done either with groups of people or specific individuals. Obtaining data often depends on serendipity (accidentally running into a terrific interviewee), or taking part in specific opportunities provided by events like conferences or observing happenings like a demonstration. Working in dangerous places is even more precarious and contingent. But a reviewer can have a sense of the project and the depth of the data based on similar work they have done in other contexts.

I am very much against posting interview transcripts. I am presently conducting field research in Algeria. If my interviewees were to think I was going to make their interviews public, even if anonymized and even if the IRB allowed it (which they won’t), they would never agree to the interviews in the first place. Even if there are no security issues involved, there are privacy issues, issues of reputation, of pride, of not wanting to malign other people needlessly, and even of libel to consider.

There are considerations of context and background which factor into one’s interpretation of one’s data. There are popular myths and stereotypes that come through in interviews and if someone not familiar with the context reads isolated interviews, they may think that the author has misrepresented a certain reality, when in fact, they have weighed various forms of evidence against one another and come to a conclusion that may contradict the interviews. It may damage relations with interviewees if one were to publicly challenge them and explain the discrepancy.

To give one example, I have done enough interviews and found enough statistical data to learn that donors and the UN did play a role in gender quota adoption in Algeria, but some activists, who were funded themselves by donors, said categorically that the donors did not play a role because they wish to provide a certain image of what happened. While I convey in my writing that it was mainly domestic actors that played the dominant role, if I were to provide the transcript of some interviews, it would show that they think domestic actors were the only actors. I know their motivation in saying this (not wanting to appear under donor influence and wanting to challenge the view that donors were the only actors) but I don’t want to challenge their view publicly and directly because these are key actors who I am sure I will interview in the future. My written work will challenge their view but only indirectly. I will cite other sources that provide what I consider a balanced perspective. However, someone reading such an interview transcript might assume I misrepresented the situation, when in fact I did not. But to explicitly explain such a discrepancy would damage my relations with my informants or it may be tangential to the study. Providing selected interview transcripts is inadequate because the reviewer does not have the full context of the interview in relation to other interviews, survey data, and other sources based on living and experiencing the situation. The interview is still only partial evidence and cannot be taken on its own as evidence of something. It has to be seen as part of a whole and not a standalone evidence in the way one might regard a statistical analysis.

So how does one evaluate qualitative evidence without replicating it? Ideally a reviewer has a familiarity with the context or a similar context. If for example we are talking about interview based evidence, in order to adequately evaluate the research, the reviewer will need to know some of the following:
• The duration, time, and general location of the study
• 1) What kinds of people were interviewed; 2) how many; 3) How representative are they of the general population or specific population if it is a targeted study; 4) do they represent a variety of demographic groupings and if not, why not and does it influence the findings; 5) do they represent a variety of views if they are elite interviews.
• How was the data analyzed.
• Limitations of the study. Possible biases in the study.
• Is there triangulation in the study? Are the findings backed up in other ways through related surveys, participant observation, other fieldwork experiences, similar research in other locations, in secondary sources, or in other primary sources such as newspaper articles? Are the findings backed up through other methods, e.g., content analysis, crossnational research, survey research. If not, what explains the discrepancy?
• What do historical or earlier studies tell us and if the pattern has changed, why has it changed?
• If the findings differ from other neighboring regions, other studies, or other contexts, can the author account for why their evidence is different.
• Remaining puzzles or unexplained findings that come from the evidence.

It is not any one piece of evidence that matters. Is is the context and how one explains how the evidence fits into that context. It is also a question of depth of knowledge of the context that is important, which is ultimately intangible.

Re: Pursuing transparency in qualitative research

Posted: Tue Nov 15, 2016 8:31 am
by Guest
My research focuses on a highly contentious topic in a quasi-democratic context. I promised all of my interviewees anonymity. If I had not been able to promise them this, most would not have even met with me. It would, moreover, be completely unethical for me to post or share my interview notes, even if I spent the time to try to anonymize them.

My question is: Does the small risk that someone is misrepresenting their interview data under existing norms outweigh the strong likelihood that certain problems and questions won't be studied and asked if we move to the proposed transparency standards?

Re: Pursuing transparency in qualitative research

Posted: Fri Dec 09, 2016 5:51 pm
by cjones
These are all excellent points. In much of my work in the Middle East, in the UAE especially but also in Kuwait and Jordan, I would also not have been able to interview a large number of subjects had I not been able to promise anonymity at the least; even with anonymity, the notion that what comes out of their mouth will be going on the internet or never dying in some other sense (e.g., entering a public repository for all academics, for all eternity) would certainly have troubled them. I would be troubled by that myself if someone wanted to interview me. Once it's out there, it's out there. It would be a good Black Mirror episode.

In all likelihood, those who *would* let me interview them would have been less useful for the research--e.g., spokespeople for educational initiatives parroting what is already on the website, rather than providing actual opinions and insights that advance knowledge. I want to highlight the point that it is not just in conflict settings where this is a problem. In a world of big data and surveillance and authoritarian backsliding and so on, many people are concerned about these issues, in democracies as well as authoritarian societies. It seems that the suggestions Aili Tripp makes (which fall short of making interview transcripts public) are very reasonable.

Re: Pursuing transparency in qualitative research

Posted: Wed Dec 21, 2016 2:31 pm
by DZS
I want to underscore and add a bit to Calvert Jones's point about the less-often discussed concern he raises about making our interview transcripts (and related evidence such as field notes, etc.) public: Doing so will almost certainly make it more difficult to earn people's trust, to make them comfortable speaking with us, to get them to allow us to observe meetings and events, etc. But the corollary to *losing* those potential interviewees is that, in addition to making it more likely that people will “parrot official positions,” those who *will be* willing to speak with us, who *will be* willing to allow us to observe their events, etc. are likely to be quite different from those who will refuse to allow our notes, the text of our conversations. etc. to be shared. Few “samples” are ever truly representative, and the goal is often not a “representative sample.” But even so, rules that would stack the deck even further to favour people who are, e.g., more extroverted, less fearful of repercussions, etc. seems very unlikely to produce better knowledge or insights about the political world.