II.2. Evidence from researcher interactions with human participants

Forum rules

We encourage contributors to the Discussion Board to publicly identify by registering and logging in prior to posting. However, if you prefer, you may post anonymously (i.e. without having your post be attributed to you) by posting without logging in. Anonymous posts will display only after a delay to allow for administrator review. Contributors agree to the QTD Terms of Use.

Instructions
To participate, you may either post a contribution to an existing discussion by selecting the thread for that topic (and then click on "Post Reply") or start a new thread by clicking on "New Topic" below.

For instructions on how to follow a discussion thread by email, click here.

Alan Kuperman
LBJ School, U. of Texas
Posts: 2
Joined: Fri Nov 18, 2016 5:06 pm

Transparency may add noise, obstruct valid inference

PostMon Dec 26, 2016 2:01 pm

I have a somewhat different concern about the proposal to require that interview transcripts be made public – ostensibly to promote cumulation of knowledge by enabling verification by future researchers – because I fear it could have the opposite effect as explained below.

My own interview research has been mainly with elites in recent or ongoing internal conflicts – i.e., the political and military elites of rebel/revolutionary/liberation movements and of the states they oppose. In my experience, the patterns of frankness of interviewees fall into three broad categories:

- Very frank from the start of the interview. This is typical of older, retired officials who have little to gain or fear.
- Initially reticent and parroting party line, but over time, as the interviewer demonstrates expertise and sympathy, becoming more frank.
- Parroting party line and refusing to open up throughout. This is typical of current and aspiring politicians.

As a result, transcripts typically are a mix of the following:

- Truth.
- Falsehood followed by Truth.
- Falsehood.

The job of the expert interviewer is two-fold: First, to elicit as much truth as possible from the interviewees using techniques that have been explained in the academic literature and elsewhere. Second, to separate the truth from the falsehood, using a variety of techniques that rely on the interviewer’s expertise of the case and research method. For example, if I know you lied in your last assertion, then I should take your next assertion with a grain of salt. Or if I hear a change in your tone of voice, or if you start blinking rapidly, or if you abruptly end the interview, it may signal a switch from honesty. By employing such techniques, the expert interviewer is separating the signal from the noise – and thus aiding the cumulation of knowledge.

The proposed requirement to make public the transcripts of interviews would effectively add back the noise to the signal. One can imagine a future researcher looking at the transcripts – without benefit of the interviewer’s case expertise, method expertise, and face-to-face presence with the interviewee – and reaching a very different conclusion based on weighing the noise and signal equally. Indeed, the future researcher could employ an ostensibly more “scientific” method, such as computerized textual analysis, to make inferences that would be far less valid – due to the well know problem of “garbage in / garbage out.”

This may not be the biggest concern with the proposal, but it is a substantial one, because it means the proposal could have an effect opposite to its stated intention.

Post Reply