Qualitative Transparency Deliberations

[Donald Searing University of North Carolina at Chapel Hill]

After reading yesterday's Comparative Politics Digest, I think that it is the editor of World Politics who has it about right. Hall is on target as well. We are talking about implementing a system that will have very undesirable unintended consequences. It will certainly discourage people, and especially young people, and particularly people in comparative politics, from doing ambitious qualitative-quantitative/mixed research. And it will discourage people from interviewing elites and others under real conditions of confidentiality (eg, "Turn it over") and from collecting complex, time-consuming interview and other qualitative data (eg, "Turn it all over in two years"). The model looks as though it is looking to micro economics whose model is physics. I think it is misguided. We are a methodologically rich diverse discipline and that is our strength. Are some colleagues hoping we can become a "real" science rather than be a social science (which uses natural science methods to study humanistic problems) by getting rid of anything that smells of the humanities? I spent a morning reading all of this newsletter and got so worked up that I'm posting about something I would ordinarily never post about. Thankfully the current editors of the APSR seem to be bending over backwards to be accommodating. But who knows what the next APSR editors will do? Talk about incentive structures: under this regime, graduate students and new assistants in comparative politics would be absolutely bonkers (until tenure or even promotion to full) to do anything other than analyze existing quantitative data sets because any of their competitors doing just that will have an enormous advantage. Transparency yes, but with these costs? How did we get to this impasse?

Donald D. Searing
University of North Carolina at Chapel Hill

[Guest]

I would go even further--at what point do we as scholars abandon political science? Meanwhile, what do we advise PhD researchers? Emerging scholars will suffer the consequences disproportionately. Over the long term, political science will eviscerate itself before higher education writ large eliminates tenured jobs.

[Tim Buthe HfP/Technical Univ of Munich & Duke University]

Thank you, Donald and Guest, for your posts--which reinforces some of the points in the great thread started by Gary Marks about whether qualitative research is held to a higher standard. In our introduction and conclusion to the symposium in QMMR, Alan Jacobs and I argued that almost all of the same demands that are made of qualitative sources of information (especially the demands for background information about concepts and measurement) should equally apply to quantitative datasets, including pre-existing ones. Reasons include: The inferences drawn from any statistical analysis are at best as good as the measurements employed to generate the data--and the suitability of many statistical techniques depends upon whether its assumptions about the data-generating process are met. My impression is that the best datasets provide this kind of meta-data, and their creators should get (and often do get) a lot of credit for it. I wonder whether it wouldn't lead to better research to insist that users of existing datasets dig up and provide such information, too (thus holding all research, including quantitative, to higher standards) than to reject the demands for more information about qualitative research?

The key questions then are: What is the background information that "transparent" research really must provide, even if costs are high? And which additional information is unreasonable to demand since the costs clearly exceed the benefits? The answers to those questions almost surely depend upon the specific kind of qualitative research a scholar is considering. It would be great if a large number of you were to weigh in concerning specific kinds of research, so as to help inform the decision how to structure the more differentiated deliberations during stage 2 of the QTD.

[Anne Pitcher]

I strongly support initiatives to make the research process more transparent and to make the evidence used to support a claim more available to scholars. The availability of generous funding, technical support, and research assistants who could scan archival data and fieldnotes; de-identify respondents to interviews, surveys, and focus groups; translate reports from Portuguese and French into English (or vice versa) would be very welcome. It would also be most useful if journal editors would extend the limits on article word length to allow qualitative scholars to more extensively document the wealth of sources they rely on to produce their work. How many of us have had to cut sources to fit the word length of a journal?

But in line with the arguments of this thread, what I find troubling about DA-RT is that it appears to be another effort to privilege the "science" in political science over the "political", to stress the "method" of doing political science over the big ideas that the methods and the evidence support. It sends a signal and a wrong one in my view that political science is exactly like studying the effects of smoking. "Data" needs to be "transparent" so it can be "replicated" in order to test another "hypothesis".

Yet much of what I find interesting in political science is not the methods but the big ideas about politics-Barrington Moore's claim regarding "no bourgeoisie, no democracy"; Jim Scott's ideas about "high modernism" and the state; Theda Skocpol's theory of revolutions; Sid Tarrow's ideas on contentious politics and social movements. Each one of these scholars has "data" to support his/her claims, but more importantly each scholar is advancing a theory, an interpretation about a series of events or a process that seeks to inform our views about the practice and the consequences of politics. One wonders if any of these works could have been produced had DA-RT been strictly enforced.

Being reminded by other scholars, journals and the association that we have an obligation to produce scholarship in accordance with the highest ethical standards of the discipline is both worthy and important. But let's remember that interpretation, perception, serendipity, experience are also integral parts of the study of politics. Anne Pitcher, University of Michigan

[Karen Alter Northwestern University]

Here is a list of unintended consequences that I have discovered while publishing in law journals, that do rely on active citation:
1) the copy-editing in law journals is 75-100% about tracking and checking citations. It is easier for editors to focus on this technical side, but it comes at the cost of making sure that the argument is clear.
2) I actually compared footnotes in law to footnotes of my polisci publications. Depending on the detail the journal wants (e.g. replication of quotes informing the inference for example), active citation adds footnotes add 25-39% to the text.
3) I already posted about concerns about exacerbating inequalities based on access to research assistants. Less resourced faculty, and those for whom number of publications is important, will be more likely publish in journals that do not require DA-RT. We might exacerbate hierarchies that already exist based on wealth of the University/college.
4) I also worry about the creeping expectations about number of publications. If a qualitative publication requires 20% more time and effort to deal with footnotes (a low estimate I think), then over time quantitative scholars could publish something like 20% more. We already know of publication rate disparities by gender, and the citation rate disparities. I anticipate that active citation will enlarge these disparities.

[William J. Kelleher, Ph.D. Independent Scholar]

Other unintended consequences of DART are that in favoring numbers, research results are necessarily superficial. Numbers can’t get into the motivations of actors, or interpret the different meanings that policies and events have for people. In this sense, numbers compel triviality. Studies have shown that the repetition of trivial research results in journals causes contempt for the political science profession among political professionals.

I discuss this in detail in my paper on Clarke and Primo at
https://independent.academia.edu/WilliamJKelleherPhD
(free safe download)

William J. Kelleher, Ph.D.

[Guest]

I'm a junior faculty member at a research university, and I've often relied on interviews with participants in the policymaking process (in the United States and elsewhere) to support in-depth case studies. It is clear to me that the unintended consequences of the proposed changes are likely to be profound. Others have made the point that these standards are likely to deter grad students and junior scholars from engaging in time-consuming qualitative research. The other side of the coin is that potential interviewees will simply decline to be interviewed. If the interview is on the record, the interviewee will not want a transcript of the interview (or the notes from it) made public. These individuals often worry about how they will be represented and how well they have articulated their points. For this reason, I am frequently asked by interviewees to consult with them before attributing a quote or a point to them in a publication. If the interview is off the record, releasing a transcript or the notes from the interview runs a risk of revealing who the individual is, even if their name is withheld, particularly if the interview discloses information that relatively few individuals would possess. Under these circumstances, I can imagine many important individuals simply declining to be interviewed - or being so cautious in their remarks that the value of the interview is greatly diminished.

[Derek Beach University of Aarhus]

I can second the problem with interviews. Having extensively interviewed elites in high-level negotiations in the EU, I can confirm that none of my interviewees would have consented to having the material archived. Many only allow me to refer to them as 'high level Commission official', if at all (many were background only). If I had to submit my interview transcripts as a precondition to publishing, I would have been forced to rely on secondary news accounts like the FT (that sometimes were correct, sometimes not).

A better way forward is to make the research process transparent so that other scholars could go out and replicate the research by talking to the same or similar people to hear whether they are getting similar accounts.

[Ches Thurber University of Chicago]

Let me start with a note of agreement saying that I agree with the possibility and undesirability of the negative consequences described above. It seems that any standard for transparency in qualitative research should seek to avoid these outcomes. Furthermore, this thread highlights the difficulty of the endeavor for qualitative research as opposed to quantitative research, where DA-RT really just codifies what had already become standard best-practices for that methodology (and explaining why quantitative scholars find it so bewildering that other political scientists find DA-RT to be at all problematic).

That said, it seems to me that the discussion of whether there can and should be some common standards for transparency in qualitative research is well worth the time. Many of the participants on these boards have been leaders in the attention they give to their sources, approaches, and transparency in their work. So the discussion need not be one of creating new rules to bring qualitative research to a higher standard, but about what are some of the best practices already being employed that could either a) be acknowledged as such and more widely employed, and b) be made easier to employ through better infrastructure and resources. For example, the creation of an active citation protocol with an easy to use repository could make it easier for more scholars to use such an approach if it fits their research design. It does not have to follow that active citations be imposed as a requirement on all qualitative scholars, or that the creation of such infrastructure needs to be opposed out of fear that its existence could lead to the de facto creation of such a requirement.

Finally, I would like to speak directly to what I see as two different types of concerns being raised:

1) Transparency standards will create issues related to copyright and protection of human subjects.

This seems clear-cut to me. We cannot impose standards that would require scholars to either violate copyrights (or in any way disseminate materials to which they do not have the rights) or disclose information that could endanger human subjects. Scholars need to have the ability to decline to provide additional "data" on these grounds and not fear being penalized by reviewers of editors for lack of transparency.

2) Transparency standards will undermine the the kind of knowledge that comes from just spending a lot of time in a country, observing and having casual conversations with people.

This argument, to me seems a little more problematic. Of course, time spent in a country, linguistic knowledge, local expertise, access to social networks, are all valuable tools for the qualitative researcher. But the value should come unique exposure to evidence that the scholar can then leverage in support of an argument. Arguments backed only by a scholar's claims of unique personal expertise are, in my opinion, a type of practice that we might want to discourage as our discipline evolves. At the very least, a discussion of the costs and benefits of this practice, or maybe a contextualization of the conditions under which this is more or less appropriate would be helpful.

[Guest]

I agree fully that the unintended consequences are likely to be profound and that this initiative favors and is designed with a certain kind of quantitative research in mind. The initiative is wrong-headed, at least in its current form.