Qualitative Transparency Deliberations

[Mala Htun Univ of New Mexico]

We also need to ask: 1) How will data access requirements respect human subjects protections? JETS states: “If cited data are restricted…authors must notify the editor at the time of submission. The editor shall have full discretion to follow their journal’s policy on restricted data, including declining to review the manuscript or granting an exemption with or without conditions.” As the letter to JETS editors signed by some 15 scholars who attended a 2015 pre-APSA workshop on DA-RT puts it, this phrase “gives license to editors to refuse to review work that follows long-established mandates to protect human subjects….[and] creates incentives for authors seeking publication to violate their commitments to protect human subjects, while assigning the cost of such violations to the author, not to reviewers or editors.” Although proponents of DA-RT have insisted that no editor will ask an author to violate IRB requirements, cases already exist in which reviewers and editors have asked authors to make available their confidential transcripts and field notes as a condition of publication (Parkinson & Wood, 2015).

In addition: 2) How can researchers uphold ethical obligations to their subjects that go beyond IRB requirements, particularly in authoritarian regimes or situations of political violence (Monroe, 2015; Parkinson & Wood, 2015; Shih, 2015; Tripp, 2015)? Shouldn’t editors and reviewers recognize that researchers are in the best position to make ethical judgments about data access?

[ReginaBateson]

Excellent points, Mala. Thank you for starting our discussion.

Of course I agree that subjects in conflict settings and authoritarian regimes can face especially grave dangers. But I worry about carving out exceptions to DART that only apply to research that poses a clear, direct risk of physical harm to subjects. As Wood and Parkinson point out, it is impossible to know what information might endanger subjects in the future, when circumstances have changed in ways that a researcher did not envision.

Furthermore, having done in-depth interviews and participant observation in both violent and non-violent settings, I think we also need to take seriously non-physical threats to subjects in more stable, peaceful settings. I don't want to see us to create a "hierarchy of subjects" or a "hierarchy of data," where some data (collected in peaceful settings) must be shared, while other data (from conflict zones) can be kept confidential.

For example, in some of my interviews in New York City, subjects have shared very personal stories about the deaths of their children and other relatives, subsequent conflicts within their families, and mental health issues. The disclosure or distribution of these full interview transcripts could obviously have dire personal, social, and professional costs for the subjects. So although bullets may not be flying, subjects who share intimate details about their lives still deserve our utmost respect and protection. A relative lack of conflict or political violence should not diminish their rights.

[Elisabeth Jean Wood Yale University]

I have many concerns about the DA-RT/JETS statement, but will focus here on implications for the protection of human subjects.

I too am deeply troubled that the editors who signed the statement believe that it is the editor -- rather than the researcher -- who should judge which data concerning human subjects must be shared with reviewers and posted online. This is not an abstract concern. In several instances, reviewers have insisted on seeing sensitive human subjects material, and editors did not deflect these requests. (In others, editors eventually supported the researcher.) It is the researcher who knows best the context and the threat that inadvertent disclosure might pose. Moreover, in the APSA Guide to Professional Ethics in Political Science, it is the individual researcher who is responsible for the ethical conduct of research.

I note that two possible ways to address this first problem are not adequate.

First, what if editors were to require the researcher to share with reviewers and post all data that is not explicitly named as confidential in the researcher’s IRB protocol? That is problematic because the procedures approved by the IRB at the beginning of a project should always be understood as the minimum of ethical requirements, not the total. Contexts change, subjects sometimes misjudge the consequences of disclosure, protection of data repository sites is uneven at best, and so on. The researcher is best positioned to make the necessary, contextual judgments.

Second, what if editors were to require that qualitative data be de-identified? That would render some qualitative data meaningless (it is the specificity and richness of the data that is the source of insight) and would pose an extreme burden on the researcher in many cases.

I also want to draw the reader’s attention to the just published Comparative Politics Newsletter, which has a symposium on DA-RT that includes source documents, essays by critics, a response by DA-RT advocates, and reflections by editors (particularly excellent is that by Deborah Yashar). See http://charlescrabtree.com/files/newsletter_spring2016.pdf.

Elisabeth Jean Wood
Yale University

[Regina Bateson MIT]

Building on Elisabeth Wood's excellent post above, I want to raise another concern about over-reliance on IRBs. IRBs do not address some very real ethical issues in our work (such as the protection of individuals and groups who are not "human subjects"). And even when we are only discussing human subjects, IRBs vary considerably. Some universities have IRBs that are tailored to the social sciences and the humanities; others do not. Procedures and standards are also diverse. Some IRBs may be quite lax in deeming political science research to be "minimal risk" or "exempt" because it does not involve direct, physical manipulation of human subjects. So as Wood argues above, researchers should have the discretion to hold themselves to more stringent standards than those required by their IRB.

Regina Bateson
MIT

[Jessica Teets Middlebury College]

I want to echo Elisabeth's concerns. I share these, especially conducting research on local officials in China today under the conditions of the on-going anti-corruption strike-hard campaign. The penalty for a guilty verdict is death, and it is unclear to me (and many officials) what triggers these investigations. It is difficult for me to build the trust I need to interview under these conditions, and one of the only things that I can really point to as protection is my ability to keep information anonymous. I explain how I don't even link my transcripts with any identifying information (location, date, time, etc) in case I was followed when meeting with them, and how I try to aggregate information rather than present specific stories and other information that might identify someone. This lack of transparency is the only protection I can offer to a local official, and having to then decide at a later date to reveal what I have promised not to (even to another scholar) or not publish my research seems like it would diminish our ability to conduct research precisely in the areas of the world where we don't have a lot of knowledge. These sources are hard to access, and anonymity is one of the only protections I can offer to someone willing to speak with me, so losing this (or not knowing exactly what I can promise to keep anonymous) will greatly hinder my ability to continue my research. Unless, of course, I am willing to no longer publish in political science journals!

[Sheena Greitens University of Missouri]

I'd like to echo the concerns raised above. Having read the DA-RT guidelines carefully, I do not believe they adequately address the requirements for protection of human subjects and others that we might encounter in the course of research. There are a couple of key points I'd like to add to what's already been stated:

1) The language of DA-RT overlooks the fundamentally important point that political scientists often have equal ethical obligations to people who are simply not covered by the IRB's narrow definition of "subjects." These could include research assistants, interpreters, archivists, colleagues at host institutions, etc. Reliance on an IRB protocol (or the phrase "human subjects") to judge whether transparency is appropriate is incredibly problematic on these grounds alone.

2) I have at times given interviewees more protection than required in my written protocol because of unexpected changes to the political conditions in my country of research. Again, reliance on what's written in an IRB to assess whether information can and should be made "transparent" is problematic because it is a minimum, often not what is safe or ethical.

3) I often do not know what information in my notes might compromise a subject. If a scholar is under surveillance, even the date of a meeting in the notes could be enough to compromise their interviewees' identity. So might a story that I do not recognize as uniquely identifying information. In many research contexts, there is simply no way to ensure confidentiality unless notes are redacted so much that they are equivalent to the information already provided in the text, in which case the value added of notes provided via the DA-RT process is negligible at best.

4) Asking editors to assess whether the protocols followed for a project places them in the position of rendering judgment on a process that is required of universities and scholars under federal law. Has anyone at DA-RT/APSA examined the legal implications of putting this burden on editors? If I were an editor, I'd be extraordinarily nervous about this.

5) Many journal editors do not work with human subjects, field research, or qualitative data. (I checked the list of journals that had signed on as of the last APSA meeting, and the percentage of editors who appear to do so is in single digits.) This heightens the risk that editors will not understand the issues raised above. I think this also places an as-yet undiscussed ethical responsibility on journal editors to educate themselves to ensure that they are adequate gatekeepers; yet I see no discussion of how this standard will be achieved.

[Guest]

I would like to second all of the above and stress the volatile and changing nature of many of our research sites. When scholars are researching in environments of ongoing violence and insecurity, it is not uncommon for a topic, venue or a process to be deemed perfectly safe one day, and to pose immense risks to research subjects, interlocutors and RAs, another day. Since security situations can change quickly, when researchers from the global north are conducting research in environments they are not intimately familiar with, I would veer towards a policy that precludes any researcher from posting transcripts or identifiers, no matter how safe they are deemed to be in the moment. Even the most diligent and responsible of researchers cannot predict the future, no matter how well they know their fieldsite. It strikes me as unfair that researchers are placed in the position of having to make decisions that might compromise the security of their local colleagues at a later date, and this problem is compounded when they are receiving competing messages and incentive structures from the profession. While many researchers are extremely diligent, I have seen many who are not. I really worry that if incentives to make data available are in place, we will see irresponsible PhD students risking the security of local staff and research subjects, either without knowing they are doing so, or without fully considering the potential consequences. The repercussions of many of the actions of foreign academics in sights of violence can often be hard to see unless you have spent extended periods of time in the field.

While I agree with all that has been said above with regard to trusting researchers rather than editors on security questions, we also need to take seriously the fact that inexperienced graduate students are increasingly encouraged to undertake fieldwork in contexts they know little about. If we are to develop professional guidelines regulating data access at all, these should first and foremost emphasize the principle of "do no harm" and ensure that even those who do not know what they are doing are not encouraged or incentivized in any way to place the security of their subjects and support staff at risk.

[Sarah Parkinson Johns Hopkins University]

Is one of the implications that emerges from this thread that scholars can't adequately address the concept of transparency without fostering a sound ethical foundation across the discipline? I share the guest poster's concern with graduate students' (and other scholars') abilities to make ethical and sound choices on these matters when it comes to human subjects research (whether in New York or South Kivu) given competing imperatives. I also note Greitens's comment that not everyone is trained on field ethics. But it seems to me that each of these comments speaks to a much broader issue regarding the state of disciplinary ethics that hasn't been deeply addressed by these discussions (as of yet).

[Maiah Jaskoski Northern Arizona University]

Data Access

My comments are based on my field research experiences. I discuss my access to human-subject and non-human-subject data, and sensitive and non-sensitive data.

I understand that, among other things, journals signing the JETS document require (1) that by the time an article is published, the author makes his or her sources available to other researchers through a digital repository, and (2) that if the author wishes to obtain an exception to this rule, he or she must request that exception from the journal at the manuscript submission stage.

I believe that I could not have complied with this standard and also accessed a great deal of data that have been crucial for my research up to this point.

Central here is a specific type of transparency that is important in my work: a commitment to telling my contacts in the field how I will manage the human-subject and non-human-subject data they give me.

MY ACCESS TO DATA THUS FAR
My contacts have shared with me their personal experiences and expertise during candid interview sessions, and they have also given me documents to help me with my research. They have trusted me—a researcher they know through mutual contacts, my writing, and/or multiple conversations with me—to analyze and place into context the information. They know that I directly maintain and control access to my interview notes and to documents they give me.

MY ACCESS TO DATA UNDER A JETS REGIME
Under a JETS regime, the situation would be different. When I met with people, I would inform them that my notes from the interview and the other information they provided me would be uploaded to a digital repository and made widely accessible to others unless I received a waiver.

Even if I promised to de-identify the information prior to uploading it, with the data-sharing possibility in place, I believe my subjects would respond guardedly to my questions, knowing that the otherwise full content of their responses could be viewed out of context by members of an unknown population. I think this challenge would apply to interviews about highly technical topics, for example my discussion with a former consultant in Bolivia about the implications of alternative natural gas pipeline designs for water contamination. I would also confront the challenge when seeking information about more sensitive issues, such as transfers of material compensation from private oil companies to local Ecuadorian army units in exchange for the units’ security services.

I believe that under JETS my contacts would refrain from offering me helpful documents out of concern that those materials could be made widely available. Had JETS been in place previously, an indigenous community member and publicly known expert in consultations in Bolivia may not have shared with me detailed timelines and descriptions of community consultations about new Bolivian natural gas projects. A Peruvian officer might not have provided me statistics on insurgent attacks maintained by the military joint command. An Ecuadorian human rights organization may not have granted me access to their statistical archives on abuses. None of these sources was categorized as sensitive, formally or informally.

[glebaron]

I want to echo the concerns articulated thus far on this thread. My research focuses on forced labor, and protection of research participants is paramount. DA-RT would put pressure on, if not directly undermine, the strict ethics protocols surrounding my data collection and storage. I suspect-- for the reasons already elaborated by colleagues in this discussion-- it would leave me unable to publish my research in leading political science journals.

Genevieve LeBaron
Yale University & University of Sheffield

[Kevin Middlebrook University College London]

I wish to second the concerns raised by Mala Htun and by Elizabeth Wood, as well as to applaud their (and other colleagues') strong leadership in this crucially important debate.

One issue that has arisen in one or more of the discussion threads is how to handle interview material gathered--either for a long-term research project, or employed years later in a subsequent, related project--well before this entire debate arose in U.S. political science. The terms that a researcher offers to an interviewee (anonymity or not, for example) must always be respected, and no conscientious researcher can alter or break those conditions (by, for instance, divulging an interviewee's identity) simply because a journal editor may "now" decide that transparency is the overriding consideration. The individual researcher should always be the responsible party in such decisions.

Requiring that the researcher produce documentation stating that research has been conducted in fulfillment of a university's code of ethics (as, I believe, the current editors of the APSR now do) will not necessarily resolve the problem of long-term protection of sensitive sources for the simple reason that some researchers change institutions over the course of their careers. The current employer could not attest in good faith to a researcher's conduct years before, and a prior employer would have no incentive to produce the required documentation in a timely manner even if the evidentiary basis for doing so has miraculously survived.

Kevin J. Middlebrook
Institute of the Americas
University College London

[Zoe Marks University of Edinburgh]

I agree with the comments and concerns raised by colleagues throughout this thread. I'm particularly in agreement with Elisabeth Wood's and Regina Bateson's points about the temporal and institutional limitations of the IRB as a protective ethical firewall, and concerns around hierarchies of protection for research participants. (Also, Aili Tripp's comments elsewhere about IRB and DA-RT working at cross-purposes.)

Ideally, the system works through checks and balances, wherein the IRB, on the one hand, and journal editors/peer reviewers (and other professional institutions), on the other, help reinforce various best practices by researchers. On any given piece of work, however, the researcher is the 'executive'. It falls to us to protect our informants and other sources, to cite and acknowledge them appropriately, to build a cohesive and competent body of work, and to maintain a reputation that enables continued professional success in both our sites of research and our sites of teaching and publication. I'd like to see this better reflected in JETS/DA-RT. Access and transparency need to fit within our broader set of obligations, and not distort or commandeer any given dimension of our work.

A few 'human subjects protection' issues that have been on my mind and which I haven't seen elsewhere on the discussion board (apologies if I've missed them in the rich debates):

1. Many have noted the need to balance costs and benefits of transparency and access. Underpinning much of this discussion, but not made explicit, is just how valuable high-risk data are to our work, particularly on violence/victimization and on elites.
Some of my best, most valuable sources - both found archives and interviews - are things I could never grant reviewers full access to, for both legal and ethical reasons, to protect myself and my informants. They are almost literally smoking guns. If we expunged the highest risk - or most poignant - interviews and sources from our work to protect them from invasive DA-RT measures, the research would *disproportionately suffer*.

2. On the flip side, DA-RT has the potential to be a boon for human subjects and vulnerable populations in particular. I've long been concerned with the extractive nature of research in post-conflict settings (my field of work) and the over-saturation of certain places with fieldwork projects (e.g. refugee camps, urban areas in newly stabilised post-war zones, etc.).
In the worst cases, victims of conflict and displacement end up being repeatedly interviewed about the same topics, with little aggregation of the copious narrative data they provide ever making its way into the public domain. The same is true of quantitative data. Literally hundreds of thousands of systematic survey interviews have been collected in Eastern Congo (e.g.) and we have used an iota of the data that have been generated. This has serious ethical implications for who bears the burden of research and whether the benefits accrue proportionally to the costs.
It is unfortunate that from an ethical perspective, DA-RT seems poised to disincentivize the context-driven, lighter footprint research that should be best practice, while encouraging or facilitating the scaled up data collection efforts than can be more easily anonymised and shared.

Ideally, data access should be as committed to fostering ethical research as it is ensuring to replicable scientific standards of research. If DA-RT is to set a new standard for data sharing, it needs to demonstrate meaningful engagement with the power inequalities and unstable contexts discussed in this thread and elsewhere on the forum, and throughout months of debate. JETS, as the long-arm extension of DA-RT, demonstrates little to no engagement with how these concerns inform for many of us everything from question identification, to data collection, analysis, dissemination, and ultimately, transparency.

3. A way forward...
I agree with Sarah Parkinson's summative contribution on political violence-related issues (16 May), which notes that many scholars address ethics, access, and transparency [EDART?] issues through a corpus of work and not just in a single article (meaty footnote/data repository/narrative appendix/etc.). The question we ask in reading each other's research is not whether we can personally access each datum pointing toward the author's claims, but whether the claims made could reasonably follow from the known/available evidence and methods.
It seems what's needed is more explicitly articulating - and cultivating - a subjective suite of best practices, rather than brittle standards, and acknowledging that at the heart of many current sub-field norms is in effect a 'balance of probability' test. Used in civil trials and for administering reparations, it applies a lower evidentiary standard than that used in criminal trials.
Others have called elsewhere for clearer articulation of minimum standards, rather than maximal best practices; a balance of probability approach would make this a sliding scale appropriate to a given piece of research. (Conveniently, this fits the claims of many DA-RT supporters that standards will be 'sensible' and not overly onerous, and that journal editors are committed to epistemic and methodological pluralism (some more than others).)