Substantive Dimensions of the Deliberations

Forum rules

We encourage contributors to the Discussion Board to publicly identify by registering and logging in prior to posting. However, if you prefer, you may post anonymously (i.e. without having your post be attributed to you) by posting without logging in. Anonymous posts will display only after a delay to allow for administrator review. Contributors agree to the QTD Terms of Use.

Instructions
To participate, you may either post a contribution to an existing discussion by selecting the thread for that topic (and then click on "Post Reply") or start a new thread by clicking on "New Topic" below.

The transition to Stage 2 of the deliberations is currently underway but will take some time to complete. In the meantime, we very much welcome additional contributions to the existing threads in this forum.

For instructions on how to follow a discussion thread by email, click here.

Sheena Greitens
University of Missouri
Posts: 8
Joined: Wed Apr 06, 2016 3:38 pm

DA-RT: effect on graduate training

PostWed Apr 20, 2016 10:14 pm

I wonder if anyone currently teaching qualitative methods has felt an impact from the DA-RT process in their graduate programs. I am currently teaching our qualitative methods course to PhD students, and my students have raised a number of questions about disciplinary requirements for archival and interview research, particularly what will be expected if they submit this work for peer review. I have referred them to the discussions over DA-RT, and tried to explain the debate fairly, while being honest/transparent with them that I do not know exactly what the discipline's standards will look like in a year or two.

My strong impression from the seminar is that the existence of this debate and the resulting lack of clarity over standards seems to *already* be having a non-trivial deterrent effect on their willingness to pursue qualitative research (or even to invest in further training in it). The peer review process seems uncertain enough to them without the added question of what a particular journal will consider sufficient transparency -- particularly when journal editors appear to have wide discretion on standards that have a large impact on them (are interview notes going to be ok? or are full transcripts the only acceptable option? how would they know ahead of time?), including over whether to accept protocols that the University's IRB has required them to follow. And given finite time and perceived high job market pressure, the extra time required to transcribe/scan/upload interview or archival material is seen as enough of an added cost that concentrating on quantitative work that can be produced more efficiently with fewer questions seems like a much better strategy for success in the field.

In other words, the uncertainty and increased transaction costs around qualitative research seem already to be leading many of them to conclude that the attempt is simply not worth the risk. I wonder if anyone else has observed a similar process at work.

Post Reply

Guest

Re: DA-RT: effect on graduate training

PostThu Apr 21, 2016 8:17 am

This is a really interesting question. Let me add my experience here just as another data point. For context, qualitative methods is required for our PhD students, and taken after they complete a two course quant sequence, and most of our students who do empirical work tend to write quant dissertations.

I assigned the DA-RT proposal and the newsletter that Alan and Tim edited last year, along with the symposium in Political Analysis on fishing and pre-registration, during the week after I had asked students to complete an exercise in active citation that used a paper they had written in a previous course. I also invited a colleague whose views on DA-RT diverge from mine to join us in order to have a debate about these issues.

In the end, since most students activated citations related to historiographical debates or their quantitative data, issues of the kind you raise regarding interview materials did not arise, and students didn't seem any less likely to pursue qualitative work than they had been initially. I wonder whether the result would be different had they been pressed to think about transparency of interviews? I may have to try that out this fall...

Post Reply

Alan Jacobs
University of British Columbia
Posts: 38
Joined: Fri Feb 26, 2016 9:59 pm

[Steering Committee] DA-RT: effect on graduate training

PostThu Apr 21, 2016 1:17 pm

Thank you, Sheena Greitens, for raising this important issue, and to the anonymous contributor for this further perspective.

It is interesting to hear that it is uncertainty about expectations that seems to be a deterrent. As I've noted in another post, many journal transparency policies (as posted online, at least) remain very general in regard to qualitative research or do not refer specifically to qualitative at all. One thing we hope these deliberations will do is to offer clearer guidance to researchers, students, and editors about what forms transparency can or should take for different forms of qualitative research.

I also want to add that we are very interested in hearing from graduate students about their experiences and concerns in regard to transparency. While we generally encourage identified posts, those wishing to remain anonymous can do so simply by posting without registering or logging in.

Post Reply

Guest

Re: DA-RT: effect on graduate training

PostMon Apr 25, 2016 4:34 am

I am a graduate student currently conducting qualitative/ethnographic field research and became aware of DA-RT after I had defended my prospectus, after I had received IRB approval, and after I had completed my first of three case studies.

With regards to the uncertainty of expectations, this is generally the case with earning a PhD in general – will what I do be good enough? At this point, perhaps my data collection and analysis plans would have been good enough pre-DA-RT, but now for many journals perhaps not. Qual methods are not required in my program, and they have been largely self-taught, trial-and-error, plus the IQMR workshops.

Doing any type of research is a big step into the unknown, making so many mistakes and only realizing our mistakes later. For my part, my early fieldnotes are patchy (particularly when I wasn't in "research mode") and some of my interviews are fairly laughable. Fortunately, I had enough time in the first case study to make up for this, and when I began my second case study, I felt far more confident, and the quality is much higher. But my first case study is still valuable – and original – and I will stand by my conclusions.

DA-RT would not have deterred me from doing qualitative research, but I do wonder how well it will be accepted by particular journals even before the peer review. A graduate degree is a learning process – with field research having a steep learning curve – and it's hard enough to have to publish before we finish our dissertations without yet another bar of uncertainty. Throughout the PhD process, the uncertainty can be almost paralyzing. I don't know what I don't know, and I've learned so much "on the job" as a researcher, but if what I don't know will come back to bite me even harder?

I'm happy to be transparent about my research process, about the mistakes I've made in my research, while keeping my research participants' information confidential. But unlike mistakes I've might have made in a choice of statistical variables or the models I might have developed, I cannot return to the conversations I had and ask for a do-over. (I have done follow-up interviews in some instances.) For someone less committed to qualitative research or someone who would only include a qualitative component to a mixed-method dissertation, I can certainly imagine skipping this (as many of my cohort-mates already do for lack of research money and time).

Post Reply

Kai Thaler
Harvard University
Posts: 2
Joined: Fri Apr 08, 2016 3:37 am

Re: [Steering Committee] DA-RT: effect on graduate training

PostTue Apr 26, 2016 2:42 pm

AlanJacobs wrote:

It is interesting to hear that it is uncertainty about expectations that seems to be a deterrent. As I've noted in another post, many journal transparency policies (as posted online, at least) remain very general in regard to qualitative research or do not refer specifically to qualitative at all. One thing we hope these deliberations will do is to offer clearer guidance to researchers, students, and editors about what forms transparency can or should take for different forms of qualitative research.


I am a graduate student currently finishing up a long period of field research. The petition to delay DA-RT and my awareness of the Journal Editors Transparency Statement (JETS) came up while I was in the field. The most worrying thing to me about the JETS and the implementation of DA-RT has been precisely the uncertainty provoked by the vagueness of DA-RT procedures. It is much easier on the surface to understand how they might apply to quantitative researchers (though others have raised issues about how true quantitative transparency might go far beyond the data and do file provision that is now standard at journals such as AJPS, JCR, and JPR), but the standards for research that draws on non-quantitative sources seems too open to ad hoc interpretation by journal editors to draw clear lessons from. Some journals already have de facto tendencies or standards for what type of research they publish, yet JETS leaves open the possibility that ten different editors could have ten different interpretations and standards, making it difficult to know what guidelines to follow and whether taking a certain research approach or drawing on a certain kind of data could foreclose the possibility of publishing in a set of journals. The different journal standards Alan Jacobs links to here are all relatively clear, but they are quite varied. I can certainly see how this would make other students think twice about pursuing research that requires data that may need to be acquired in a manner that proscribes the idealized vision of transparency, whether due to ethical concerns, cost concerns, or time concerns.

Greater clarity in broader standards and implementation would help ease concerns among many scholars, not just graduate students, and would ensure that the implications of DA-RT are better able to be considered, debated, and decided upon.

Post Reply

Guest

Re: DA-RT: effect on graduate training

PostThu May 05, 2016 5:29 am

I'm another graduate student also currently conducting field research abroad. The DA-RT initiative and debate began as I embarked on fieldwork.

-As pointed out by others in this thread, the DA-RT guideline do add more stress and ambiguity to using qualitative methods in my dissertation fieldwork. I've discussed this with fellow graduate students from my program also conducting field work abroad. The general consensus was that in the short term, it provides a strong disincentive to simply not submit to journals which appear to take a strong stance on DA-RT guidelines. There are largely concerns about time: if we are trying to submit work before going on the job market, we must now consider cleaning up and translating interview notes, preparing archival scans, preparing extra appendices. Additionally, it adds another level of uncertainty in the review process: if I decide that my field notes are too sensitive to share, will this make my work seem less "transparent," even if I give reasons for not sharing them?

To turn to the DA-RT proposals more generally:

-I support a "soft" version of transparency in social science research. Readers should have some idea of how I went about collecting the data, how I weighed competing claims, and other details of the research process (well described by Margaret Keck in another post).

The "hard" version of transparency put forward by DA-RT is based on a premise that it is productive for other scholars to have access to my mass of interview transcripts, field observations, digital recordings, and photocopied fragments from archives. What will not be shared, along with this mass of documents, is the critical eye that I took to it; the writing of field memos where I considered the meaning of the interviews. This is arguably where the real intellectual work is done, but represent the types of documents that most scholars would never consider sharing.

-What is most frustrating about the DA-RT initiative for me is the vague notion of the costs and benefits, and intended audience, of the extra appendices, active citations, and qualitative data repositories.

The critical readers of my work will not likely comb through my interview transcripts to find "gotcha" moments - ("Why did Interview #23 say x, while Interview #24 said y?"). They will likely have something to say about my research methods, my interpretation of key historical events, my characterization of the literature, and so on. Area specialists can draw upon their knowledge of the cases at hand to question my results or interpretations. Readers with a thematic interest will find other critiques to make about case selection, the evidence I use to support key claims, etc. Up to this point, editors, reviewers, and readers have generally not had trouble finding ways to critique what was in the published work. This goes for people who work on my area, area specialists, and general readers.

Who, then, would comb through my prepared, translated, and formatted interview notes? I don't imagine there are a lot of qualitative researchers who would be so bold to claim that they could try to "replicate" qualitative results by reading through another researcher's interview notes. Even in using historical archival sources - is it necessarily harder to cherry pick quotes when they are 150 words long?

Post Reply

Guest

Re: DA-RT: effect on graduate training

PostThu May 05, 2016 6:26 am

I can see a use for large repositories of interviews and field-notes in meta-studies, in quantitative analysis based on keyword frequencies, historical studies, network studies and other down-stream type of research. Before deciding to study question X in country Y it would be extremely useful for me to have access to all the questionnaires and interview protocols that were used to investigate the same question ideally in such country. So there is a potential gain in uploading all this raw data and research protocols for future secondary research, but this gain has very little to do with checking the validity of a qualitative study, it's more to inform research design, highlight pitfalls of questions that do not work and conduct new research. And this gain might be better generated by creating a unified repository of all interviews on a topic that researchers can decide to opt in, instead of 100 different small repositories with slightly different rules for tagging, uploading and storing data. Such repository could also target null finding research and thus overcome the massive publication bias against 'failed research'.

Post Reply