Page 1 of 1

Power and the Institutionalization of Research Transparency/Openness/Explicitness

Posted: Sun Sep 18, 2016 8:06 am
by TimButhe
In this thread, we especially invite discussion of how various forms of "explicitness" about one's sources, analytical approach(es), and methods interact with power and resource differentials between scholars. Do you have reason to believe (maybe based on your own experiences) to suggest certain transparency practices (or particular ways of institutionalizing them) would exacerbate particular existing inequities? Conversely, do you have reason to believe that certain forms of explicitness will empower currently disadvantaged scholars?

Re: Power and the Institutionalization of Research Transparency/Openness/Explicitness

Posted: Sun Sep 25, 2016 10:13 pm
by Guest
[quote="TimButhe"]In this thread, we especially invite discussion of how various forms of "explicitness" about one's sources, analytical approach(es), and methods interact with power and resource differentials between scholars. Do you have reason to believe (maybe based on your own experiences) to suggest certain transparency practices (or particular ways of institutionalizing them) would exacerbate particular existing inequities? Conversely, do you have reason to believe that certain forms of explicitness will empower currently disadvantaged scholars?[/quote]

One statement of this topic I have seen appears to set up a binary between "rules with enforcement" and "norms with empowering capacity-building or enaabling pedagogy."

I think it is more helpful to think in terms of 8 possibilities, which I will outline with a decision tree-type notation:

no capacity-building pedagogy 1
enforcement capacity-building pedagogypedegogy 2
rules
enabling no capacity-building pedagogy 3
capacity-building pedagogy 4

no capacity-building pedagogy 5
enforcement capacity-building pedagogypedegogy 6
norms
enabling no capacity-building pedagogy 7
capacity-building pedagogy 8

Rules need followers, and hence enabling; if rules are supported only by enforcement the enforcement gets overwhelmed; internalizing norms enables, but norms also need enforcement against those who won't internalize. So my own take is that we are really talking about a combination of 4 and 8 because people will need to learn stuff to be able to understand and follow rules and norms.

MJ Peterson
University of Massachusetts Amherst

Re: Power and the Institutionalization of Research Transparency/Openness/Explicitness

Posted: Wed Oct 26, 2016 10:35 am
by jane mansbridge
In my experience, many qualitative researchers do not have grants (or at least extensive grants). They do much of their research on their own time, using their own personal monetary resources. They may receive a travel grant, but rarely a grant that actually covers all of the expenses for a field trip, let alone the time required to write up the results. In my own case, although I have held fellowships, I have never (if my memory is right) received a grant to do any of my qualitative research.

The less well-resourced the institution the scholar works in, the more that scholar is underfunded.

Having to incur costs to convert field notes or interviews into the appropriate forms for deposit and having to pay fees for depositing these notes and interviews in a qualified depository might well prove impossible for such researchers.

I worry not only that these added costs will fall disproportionately on the least well-resourced scholars (who are often closer to settings that merit qualitative research), but also that these costs will discourage younger scholars from taking qualitative approaches. Choosing a dissertation topic, they may find it much easier and attractive just to use a database already in the public domain. This predictable dynamic discourages innovative research, making it harder for new questions to come on the political science agenda.

Re: Power and the Institutionalization of Research Transparency/Openness/Explicitness

Posted: Wed Oct 26, 2016 3:01 pm
by Tasha Fairfield
I would like to agree with Jane that this kind of requirement could serve as a significant disincentive for conducting qualitative research, which has made major contributions across all fields of political science--alongside concerns about human subjects, and the problem of access to key informants. I do not conduct research in difficult setting or on highly sensitive topics, but even so, I feel that asking for permission to make interview transcripts public would have damaged rapport and access.

jane mansbridge wrote:I worry not only that these added costs will fall disproportionately on the least well-resourced scholars (who are often closer to settings that merit qualitative research), but also that these costs will discourage younger scholars from taking qualitative approaches. Choosing a dissertation topic, they may find it much easier and attractive just to use a database already in the public domain. This predictable dynamic discourages innovative research, making it harder for new questions to come on the political science agenda.

Re: Power and the Institutionalization of Research Transparency/Openness/Explicitness

Posted: Tue Nov 15, 2016 7:48 am
by Guest
Jane Mansbridge writes: "I worry ... that these costs will discourage younger scholars from taking qualitative approaches. Choosing a dissertation topic, they may find it much easier and attractive just to use a database already in the public domain. This predictable dynamic discourages innovative research, making it harder for new questions to come on the political science agenda."

I would like to echo this concern that the costs of these new transparency standards will lead young scholars (in particular, but not only) to choose easier and lower-cost methods and questions that can be answered with those methods. This is already happening, with more and more new PhDs doing survey experiments, which can be outsourced to a survey research firm and whose external validity can be questionable, instead of spending months in one or more countries doing interviews or archival research. The latter is increasingly devalued, and the DART transparency standards promise to further accelerate this trend, to the detriment of the discipline's ability to accumulate real knowledge about complex problems.

Re: Power and the Institutionalization of Research Transparency/Openness/Explicitness

Posted: Tue Nov 15, 2016 11:20 am
by MichelleJurkovich
jane mansbridge wrote:In my experience, many qualitative researchers do not have grants (or at least extensive grants). They do much of their research on their own time, using their own personal monetary resources. They may receive a travel grant, but rarely a grant that actually covers all of the expenses for a field trip, let alone the time required to write up the results


I'd like to echo Jane Mansbridge's point above. Transcribing interviews has been a far more expensive and time consuming process than I'd expected and certainly not one I would have been able to do if I hadn't been fortunate to receive financial support from my institution. My transcriptions are not for public viewing (part of the agreement I made with interviewees in securing informed consent) and the cost and time would have been even greater if I'd needed to worry about making them viewable by the public (i.e. removing identifiers), which again, I can't do as that was not part of the agreement I made with participants. Leaving aside all ethical and human subject concerns (which are many!!!!) if journals are not going to provide the funding for this process it will place a severe financial hardship on qualitative scholars---many of whom will simply not be able to submit to these journals if there is such a high financial burden to do so.

Re: Power and the Institutionalization of Research Transparency/Openness/Explicitness

Posted: Thu Nov 17, 2016 7:50 am
by resodihardjo
I concur with those stating that the proposed policies for opennes and accessability of data favour quantitative research and make it less likely for students and reseachers to opt for qualitative research (either by design or because they no longer have access to data). It is, however, not the cost of transcribing that I think is the most problematic (as transcribing interviews is often (but not always) needed for a proper analysis of the interviews), but the access to respondents if researchers are forced to make their interviews public. After all,
(1) respondents will be less likely to agree to an interview if they know the transcript of the interview will be made public
(2) true anonimity can never be guaranteed when 'sanitizing' transcripts from personal characterizations. There was only one person who was, for example, Home Secretary when prison escape X happened so the transcript can be easily linked to the respondent.

And if respondents do agree, they might censure themselves as they know that their statements will be made public. Consequently, it will be more difficult for researchers to do their research.

Moreover, this proposal raises numerous ethical and legal questions, including
(a) questions about national data protection/privacy laws which may vary per country and thus affect researchers differently (i.e. making it impossible to publish in US journals)
(b) re-use of data even though respondents only agreed that their data could be used by the original researcher (i.e. they agreed to be interviewed for a certain research, with a specific research question)

The question is: what are we hoping to achieve by all of this? More transparancy? Better research? The follow up question is: do the proposed solutions really help to achieve this?

I'm not in favour of creating a one size fits all format for research - as it will definitely not fit all sizes. It's better to set a minimum level of requirement for all research (clear research/methodology section; proper referencing as citations can still be sorely lacking - and yes that is partially due to word limit) and then add additional information based on what is needed. So if you do qualitative research and opted for inductive content analysis of interviews, it's not the interviews that need to be published online, it is the complete codebook with an explicit explanation of what you did in each round of coding. Depending on the type of research, these codebooks can be quite lengthy. If you opted, for instance, for deductive media analysis, the codebook needs to be published online as well. In both cases, researchers need to explain in their methodology section what they did, followed by a reference to the online codebook for those who want to replicate the study.

Re: Power and the Institutionalization of Research Transparency/Openness/Explicitness

Posted: Mon Nov 28, 2016 9:13 am
by mneesha
Leaving aside all of my other myriad objections to the call to share interview transcripts, let me offer a very pragmatic testimonial. As an early career scholar at an LAC with little endowment, I already have to creatively stretch my small faculty fund to get myself to conferences each year. Several years ago when I decided to outsource transcribing a portion of my interviews for my own personal use, that meant that I couldn't afford to go to an international conference, nor to return to a research site that needed follow up, and this same trade off will be the same for future projects, unless, as someone mentioned, journals that require transcripts start paying for their production.

I also have concerns about what this financial burden of transcription means in terms of widening the inequality gap between faculty at different kinds of institutions, and with different profiles. At risk of oversimplifying, will white, male R1 faculty get the grants that allow them to "do it all," while female LAC faculty continue to make these trade offs? The "burden of proof" for research risks exacerbating an already problematic equity divide in academia. Mix this with a methodological referendum like DART, and the dangers to a certain profile to which I happen to belong (female LAC faculty doing qualitative research with ethnic minority social movement participants in sometimes politically unstable places) seem rife.


resodihardjo wrote:I concur with those stating that the proposed policies for opennes and accessability of data favour quantitative research and make it less likely for students and reseachers to opt for qualitative research (either by design or because they no longer have access to data). It is, however, not the cost of transcribing that I think is the most problematic (as transcribing interviews is often (but not always) needed for a proper analysis of the interviews), but the access to respondents if researchers are forced to make their interviews public. After all,
(1) respondents will be less likely to agree to an interview if they know the transcript of the interview will be made public
(2) true anonimity can never be guaranteed when 'sanitizing' transcripts from personal characterizations. There was only one person who was, for example, Home Secretary when prison escape X happened so the transcript can be easily linked to the respondent.

And if respondents do agree, they might censure themselves as they know that their statements will be made public. Consequently, it will be more difficult for researchers to do their research.

Moreover, this proposal raises numerous ethical and legal questions, including
(a) questions about national data protection/privacy laws which may vary per country and thus affect researchers differently (i.e. making it impossible to publish in US journals)
(b) re-use of data even though respondents only agreed that their data could be used by the original researcher (i.e. they agreed to be interviewed for a certain research, with a specific research question)

The question is: what are we hoping to achieve by all of this? More transparancy? Better research? The follow up question is: do the proposed solutions really help to achieve this?

I'm not in favour of creating a one size fits all format for research - as it will definitely not fit all sizes. It's better to set a minimum level of requirement for all research (clear research/methodology section; proper referencing as citations can still be sorely lacking - and yes that is partially due to word limit) and then add additional information based on what is needed. So if you do qualitative research and opted for inductive content analysis of interviews, it's not the interviews that need to be published online, it is the complete codebook with an explicit explanation of what you did in each round of coding. Depending on the type of research, these codebooks can be quite lengthy. If you opted, for instance, for deductive media analysis, the codebook needs to be published online as well. In both cases, researchers need to explain in their methodology section what they did, followed by a reference to the online codebook for those who want to replicate the study.

Re: Power and the Institutionalization of Research Transparency/Openness/Explicitness

Posted: Wed Nov 30, 2016 3:15 pm
by elocoman
I offer my opinion from the graduate student perspective, being at the stage of writing my dissertation: I studied archival documents in the ministries of foreign affairs of two countries and one of the main rules I had to agree to before I was granted access to the diplomatic documents was to NEVER make the documents I studied public. I worry that if I want to publish an article from my dissertation in the future, I will be asked to make the documents publicly available and this would mean that I do not follow the rules that I agreed to before I was given access to the documents. Before I was given the chance to study the diplomatic documents, I was asked to sign a “contract,” which clearly stated that I will abide by the rule to keep the materials safe, not share them publicly. In addition, I have plans to return to the diplomatic archives and continue the research for the years which I did not cover during my previous field research activity: I am worried that the foreign affairs officials would not allow me to study the diplomatic files if I tell them that I plan to make them public.

There is also the question of AVAILABILITY of documents: one site which I visited for my field research did not allow me, as a researcher, to take photos of the documents – I could only read and take written notes. If I use some of these sources and quotes in a future article, how am I supposed to show “proof” of the original document where I took the quote?

As some of the scholars above wrote, we, as researchers using qualitative tools, are put at a disadvantage in rapport with the colleagues using quantitative methods: not only it is taking me more than 12 months to read, translate and annotate all these documents from other languages into English , I have to also worry about making them available. I have colleagues in graduate school who deliberately do not use qualitative methods, because they want to graduate quickly, find a job and not spend years doing their PhDs.

Re: Power and the Institutionalization of Research Transparency/Openness/Explicitness

Posted: Fri Dec 09, 2016 5:40 pm
by Guest
I agree with previous commenters that transparency standards such as active citation require a considerable investment in terms of researchers' time. Consequently, I also agree that transparency standards such as active citation have the potential to increase the divide between the "haves" -- people who have the time to hire RAs to help them meet such requirements -- and the "have nots" in our profession.

That being said, I think that it is important to acknowledge that qualitative data sharing can also empower people in our profession (like me) who have limited access to resources for field research, archival visits, and so on. Let me give an example. Some other political scientists working in my research area conducted >100 interviews that they subsequently published online in the form of an oral history project several years ago. I -- and other researchers -- have made great use of that oral history project's interview transcripts to answer different research questions. I have also used interview transcripts housed in oral history projects produced by international organizations several other times in the past. When I haven't had the ability to get overseas because I didn't have access to enough research money -- and when I had trouble as a graduate student getting people to respond to my e-mails with requests for interviews -- this kind of data sharing really helped me.

So yes, transcribing interview notes or making archival notes available takes a lot of resources. And yes, doing that is often impossible from a ethical or legal perspective. (And yes, for researchers working in an ethnographic tradition, the idea of sharing ones interview notes to promote "replication" or "transparency" or whatever is absurd.) Yes to all of that. HOWEVER, making available interview transcripts (or archival documents) -- ONLY when feasible, ethical, and legal, of course -- can also really help under-resourced and junior scholars. This gain should also be acknowledged by those who are concerned about equity issues in our profession.

I'm not sure what the solution is. If we reward researchers who provide the public good of this kind of data access, then we are more likely to reward the researchers who are already well-resourced. Then again, if we reward researchers who provide the public good of this kind of data access, then I think more people will do it, and I do think there is something real that under-resourced scholars can gain from that kind of data access.

Re: Power and the Institutionalization of Research Transparency/Openness/Explicitness

Posted: Fri Dec 16, 2016 7:09 pm
by Vicky Murillo (Columbia University)
[quote="jane mansbridge"]In my experience, many qualitative researchers do not have grants (or at least extensive grants). They do much of their research on their own time, using their own personal monetary resources. They may receive a travel grant, but rarely a grant that actually covers all of the expenses for a field trip, let alone the time required to write up the results. In my own case, although I have held fellowships, I have never (if my memory is right) received a grant to do any of my qualitative research.

The less well-resourced the institution the scholar works in, the more that scholar is underfunded.

Having to incur costs to convert field notes or interviews into the appropriate forms for deposit and having to pay fees for depositing these notes and interviews in a qualified depository might well prove impossible for such researchers.

I worry not only that these added costs will fall disproportionately on the least well-resourced scholars (who are often closer to settings that merit qualitative research), but also that these costs will discourage younger scholars from taking qualitative approaches. Choosing a dissertation topic, they may find it much easier and attractive just to use a database already in the public domain. This predictable dynamic discourages innovative research, making it harder for new questions to come on the political science agenda.[/quote]


This is Vicky Murillo @ Columbia University (unable to login after registration):

In response to Jane Mansfield comments, which echo those of Peter Hall in the CP newsletter, I want to add another issue to the discussion.
Even if the journal or depository pays for the cost of conversion of the interviews or notes into digital format, which I believe should be the case to make it equally affordable to all scholars, it seems to me that the issue of time and the issue of contextual accuracy would still be pending.
First, contextual accuracy is crucial in the interpretation of data either quantitative or qualitative. We have talked about the obvious methods of triangulation and relying on multiple sources but sometimes even if a 3 or 4 interviews coincide, these may not be the adequate person to interview or all of them may be misrepresenting reality. It seems to me that beside the obvious check of blind review in journals, there should be some mechanisms to involve reviewers from the region when the relevant work is on comparative politics. Reviewers from the region would hopefully be less tied by internal academic networks and more akin to the interpretation of whether the choice of actors to be interviewed or their representation of reality may be wrong. Currently, American academy relies on data (quantitative and qualitative) collected by US trained scholars and we have to accept that is true even when local scholars may have made very different choices in terms of which data is relevant.
Second, the one year limit in turning qualitative data for repositories is not reflective of the real time it takes to analyze this data given other obligations in my personal experience. It seems to me that having finished the project (book for instance) is the time that is right for the scholar to make his/her data available (and as I said the cost should be paid by repositories of making the data digital) and it is in her/his interest to do it so that others can cite his/her work and check his/her findings to contribute to the scholarly debate.

Re: Power and the Institutionalization of Research Transparency/Openness/Explicitness

Posted: Fri Dec 30, 2016 6:49 pm
by RobinLTurner
As a faculty member at a teaching-intensive university, I agree with the concerns expressed by Jane Mansbridge and other commentators regarding the severe, differential burdens DART-style transparency practices would impose on qualitative researchers, those at less well-resourced institutions, and non-tenure-track/independent scholars. The time and/or money required to transform the data I generate through interviews, meeting observation, and other field interactions into transcripts readily accessible/transparent to scholars and other people is prohibitive. As one guest wrote (http://tinyurl.com/jzmbzmp), "This rule turns the dollar cost of our entire project into the cost of a single article." The university start up and internal competitive grants that have funded my field research trips to southern Africa would not cover DART-related costs.

These data transparency costs are far from proportionate given the limited utility of a word-for-word transcript read without understanding not only of the research context (see Vicky Murillo, this section, Aili Tripp II.C, http://tinyurl.com/gwtox3v) but also of the particular interpersonal interaction between that researcher and participant (see Tim Pachirat, I.1, http://tinyurl.com/j9wp6a2).I see research as "a specifically relational and deeply intersubjective enterprise" (Tim) in which no data is truly raw or unfiltered. Present disciplinary norms call for scholars to state "what work we've done, and how" (Amanda Fulmer, II.2, http://tinyurl.com/z6wcjc4) but don't require everyone to address positionality. I see this disregard as a greater threat to valid research inference than narrowly defined data transparency (my work in progress).

Like many QTD deliberation participants, I am deeply concerned that DART will harm qualitative inquiry by further marginalizing qualitative/interpretive scholarship and damaging the interactions with research subjects/participants through which many political scientists generate data.

Re: Power and the Institutionalization of Research Transparency/Openness/Explicitness

Posted: Sat Dec 31, 2016 11:44 pm
by Tim Luke
The overall DA-RT process provides many grounds to to believe certain transparency practices, and the particular ways of institutionalizing them via the declarations of political science journal editors with the APSA's approval will continue to deepen existing inequities in political science. Indeed, how far away from a regimen of accreditation along the lines of architects, lawyers, physicians, or urban planners is the DA-RT process leading? There is little reason to believe certain forms of contesting transparency could empower currently disadvantaged scholars. Instead the trends are pointing in opposite directions, namely, towards mainstream norms of professional correctness becoming institutions for enforcing officially correct professionally research that require periodic reaffirmation via "continuing methodological education,"
"correct transparency practices," or "examination board certification" of how "good science" is performed with the discipline
as recognized by professional organizations, associations for college/university accreditation, and state councils of education.
Once again, the desires of many anxious social scientists all to eager to prove their equality with biomedical, natural or
physical sciences could be affirmed by requiring members in the discipline to submit to these trials of legitimacy.