Substantive Dimensions of the Deliberations
We encourage contributors to the Discussion Board to publicly identify by registering and logging in prior to posting. However, if you prefer, you may post anonymously (i.e. without having your post be attributed to you) by posting without logging in. Anonymous posts will display only after a delay to allow for administrator review. Contributors agree to the QTD Terms of Use.
Instructions
To participate, you may either post a contribution to an existing discussion by selecting the thread for that topic (and then click on "Post Reply") or start a new thread by clicking on "New Topic" below.
The transition to Stage 2 of the deliberations is currently underway but will take some time to complete. In the meantime, we very much welcome additional contributions to the existing threads in this forum.
For instructions on how to follow a discussion thread by email, click here.
-
Sheena Greitens
University of Missouri - Posts: 8
- Joined: Wed Apr 06, 2016 3:38 pm
DA-RT: effect on graduate training
My strong impression from the seminar is that the existence of this debate and the resulting lack of clarity over standards seems to *already* be having a non-trivial deterrent effect on their willingness to pursue qualitative research (or even to invest in further training in it). The peer review process seems uncertain enough to them without the added question of what a particular journal will consider sufficient transparency -- particularly when journal editors appear to have wide discretion on standards that have a large impact on them (are interview notes going to be ok? or are full transcripts the only acceptable option? how would they know ahead of time?), including over whether to accept protocols that the University's IRB has required them to follow. And given finite time and perceived high job market pressure, the extra time required to transcribe/scan/upload interview or archival material is seen as enough of an added cost that concentrating on quantitative work that can be produced more efficiently with fewer questions seems like a much better strategy for success in the field.
In other words, the uncertainty and increased transaction costs around qualitative research seem already to be leading many of them to conclude that the attempt is simply not worth the risk. I wonder if anyone else has observed a similar process at work.
Post Reply
-
Guest
Re: DA-RT: effect on graduate training
I assigned the DA-RT proposal and the newsletter that Alan and Tim edited last year, along with the symposium in Political Analysis on fishing and pre-registration, during the week after I had asked students to complete an exercise in active citation that used a paper they had written in a previous course. I also invited a colleague whose views on DA-RT diverge from mine to join us in order to have a debate about these issues.
In the end, since most students activated citations related to historiographical debates or their quantitative data, issues of the kind you raise regarding interview materials did not arise, and students didn't seem any less likely to pursue qualitative work than they had been initially. I wonder whether the result would be different had they been pressed to think about transparency of interviews? I may have to try that out this fall...
Post Reply
-
Alan Jacobs
University of British Columbia - Posts: 38
- Joined: Fri Feb 26, 2016 9:59 pm
[Steering Committee] DA-RT: effect on graduate training
It is interesting to hear that it is uncertainty about expectations that seems to be a deterrent. As I've noted in another post, many journal transparency policies (as posted online, at least) remain very general in regard to qualitative research or do not refer specifically to qualitative at all. One thing we hope these deliberations will do is to offer clearer guidance to researchers, students, and editors about what forms transparency can or should take for different forms of qualitative research.
I also want to add that we are very interested in hearing from graduate students about their experiences and concerns in regard to transparency. While we generally encourage identified posts, those wishing to remain anonymous can do so simply by posting without registering or logging in.
Post Reply
-
Guest
Re: DA-RT: effect on graduate training
With regards to the uncertainty of expectations, this is generally the case with earning a PhD in general – will what I do be good enough? At this point, perhaps my data collection and analysis plans would have been good enough pre-DA-RT, but now for many journals perhaps not. Qual methods are not required in my program, and they have been largely self-taught, trial-and-error, plus the IQMR workshops.
Doing any type of research is a big step into the unknown, making so many mistakes and only realizing our mistakes later. For my part, my early fieldnotes are patchy (particularly when I wasn't in "research mode") and some of my interviews are fairly laughable. Fortunately, I had enough time in the first case study to make up for this, and when I began my second case study, I felt far more confident, and the quality is much higher. But my first case study is still valuable – and original – and I will stand by my conclusions.
DA-RT would not have deterred me from doing qualitative research, but I do wonder how well it will be accepted by particular journals even before the peer review. A graduate degree is a learning process – with field research having a steep learning curve – and it's hard enough to have to publish before we finish our dissertations without yet another bar of uncertainty. Throughout the PhD process, the uncertainty can be almost paralyzing. I don't know what I don't know, and I've learned so much "on the job" as a researcher, but if what I don't know will come back to bite me even harder?
I'm happy to be transparent about my research process, about the mistakes I've made in my research, while keeping my research participants' information confidential. But unlike mistakes I've might have made in a choice of statistical variables or the models I might have developed, I cannot return to the conversations I had and ask for a do-over. (I have done follow-up interviews in some instances.) For someone less committed to qualitative research or someone who would only include a qualitative component to a mixed-method dissertation, I can certainly imagine skipping this (as many of my cohort-mates already do for lack of research money and time).
Post Reply
-
Kai Thaler
Harvard University - Posts: 2
- Joined: Fri Apr 08, 2016 3:37 am
Re: [Steering Committee] DA-RT: effect on graduate training
AlanJacobs wrote:
It is interesting to hear that it is uncertainty about expectations that seems to be a deterrent. As I've noted in another post, many journal transparency policies (as posted online, at least) remain very general in regard to qualitative research or do not refer specifically to qualitative at all. One thing we hope these deliberations will do is to offer clearer guidance to researchers, students, and editors about what forms transparency can or should take for different forms of qualitative research.
I am a graduate student currently finishing up a long period of field research. The petition to delay DA-RT and my awareness of the Journal Editors Transparency Statement (JETS) came up while I was in the field. The most worrying thing to me about the JETS and the implementation of DA-RT has been precisely the uncertainty provoked by the vagueness of DA-RT procedures. It is much easier on the surface to understand how they might apply to quantitative researchers (though others have raised issues about how true quantitative transparency might go far beyond the data and do file provision that is now standard at journals such as AJPS, JCR, and JPR), but the standards for research that draws on non-quantitative sources seems too open to ad hoc interpretation by journal editors to draw clear lessons from. Some journals already have de facto tendencies or standards for what type of research they publish, yet JETS leaves open the possibility that ten different editors could have ten different interpretations and standards, making it difficult to know what guidelines to follow and whether taking a certain research approach or drawing on a certain kind of data could foreclose the possibility of publishing in a set of journals. The different journal standards Alan Jacobs links to here are all relatively clear, but they are quite varied. I can certainly see how this would make other students think twice about pursuing research that requires data that may need to be acquired in a manner that proscribes the idealized vision of transparency, whether due to ethical concerns, cost concerns, or time concerns.
Greater clarity in broader standards and implementation would help ease concerns among many scholars, not just graduate students, and would ensure that the implications of DA-RT are better able to be considered, debated, and decided upon.
Post Reply
-
Guest
Re: DA-RT: effect on graduate training
-As pointed out by others in this thread, the DA-RT guideline do add more stress and ambiguity to using qualitative methods in my dissertation fieldwork. I've discussed this with fellow graduate students from my program also conducting field work abroad. The general consensus was that in the short term, it provides a strong disincentive to simply not submit to journals which appear to take a strong stance on DA-RT guidelines. There are largely concerns about time: if we are trying to submit work before going on the job market, we must now consider cleaning up and translating interview notes, preparing archival scans, preparing extra appendices. Additionally, it adds another level of uncertainty in the review process: if I decide that my field notes are too sensitive to share, will this make my work seem less "transparent," even if I give reasons for not sharing them?
To turn to the DA-RT proposals more generally:
-I support a "soft" version of transparency in social science research. Readers should have some idea of how I went about collecting the data, how I weighed competing claims, and other details of the research process (well described by Margaret Keck in another post).
The "hard" version of transparency put forward by DA-RT is based on a premise that it is productive for other scholars to have access to my mass of interview transcripts, field observations, digital recordings, and photocopied fragments from archives. What will not be shared, along with this mass of documents, is the critical eye that I took to it; the writing of field memos where I considered the meaning of the interviews. This is arguably where the real intellectual work is done, but represent the types of documents that most scholars would never consider sharing.
-What is most frustrating about the DA-RT initiative for me is the vague notion of the costs and benefits, and intended audience, of the extra appendices, active citations, and qualitative data repositories.
The critical readers of my work will not likely comb through my interview transcripts to find "gotcha" moments - ("Why did Interview #23 say x, while Interview #24 said y?"). They will likely have something to say about my research methods, my interpretation of key historical events, my characterization of the literature, and so on. Area specialists can draw upon their knowledge of the cases at hand to question my results or interpretations. Readers with a thematic interest will find other critiques to make about case selection, the evidence I use to support key claims, etc. Up to this point, editors, reviewers, and readers have generally not had trouble finding ways to critique what was in the published work. This goes for people who work on my area, area specialists, and general readers.
Who, then, would comb through my prepared, translated, and formatted interview notes? I don't imagine there are a lot of qualitative researchers who would be so bold to claim that they could try to "replicate" qualitative results by reading through another researcher's interview notes. Even in using historical archival sources - is it necessarily harder to cherry pick quotes when they are 150 words long?
Post Reply
-
Guest
Re: DA-RT: effect on graduate training
Post Reply