Qualitative Transparency Deliberations

[Kimberly Morgan George Washington University]

As someone who uses archival and other textual sources, I would love to have some more discussion about active citation as a way forward, or backward, when it comes to increasing transparency in qualitative research. The examples at the Qualitative Data Repository reveal some benefits of having scholars show what evidence they used to reach their conclusions, but it also appears that this requires a large time investment (not to mention problems of meeting this standard when archives do not allow their materials to be publicly available, as Michelle Jurkovich noted in another thread).

So my questions are the following: (1) is active citation something that should be promoted, or refused? (2) If the latter, are there other ways scholars can strive to increase transparency, such as through an online appendix that spells out the choices made with regard to data selection, and how the author drew inferences from her/his sources? (3) Can better footnoting practices address some of the complaints of those who say qualitative researchers are not sufficiently transparent? Historians, for instance, use meaty footnotes to display the sources used or address conflicting points or debates over the meaning of evidence. Could one way forward be to develop a set of standards with regard to the citation and discussion of qualitative evidence in footnotes?

If new transparency standards are to be required of qualitative researchers in journal publishing, it strikes me that qualitative researchers deserve to get something in return -- longer word counts, the ability to have lengthy footnotes, etc., so that we could actually meet these standards. I'd be interested to hear what people think of that: what would we, as scholars, need to meet these standards?

Click below to read comments that colleagues have made so far about active citation and working with documents:

-Summary post on: what does data sharing concretely mean?
-Re: Against “requirements”
-Re: [From Steering Comm.] Inviting input on specific transparency practice
-DA-RT: effect on graduate training
-[From Steering Comm.] What might qualitative data access look like?
-Re: Are qualitative researchers held to a higher standard?

[Chloe Thurston Northwestern University]

As someone else who uses archival and textual sources, I really like these questions/suggestions. I think (2) increasing transparency through means like online indices, etc. and (3) better footnoting, would address the transparency problems better than active citation. Except in cases where entire collections are already digitized I am not sure I see the value in active citation as the standard just yet (you exchange a lot of work and a legal gray area with regards to copyright/fair use for decontextualized portions of a manuscript – though sometimes a sentence or two really is all that’s necessary to give the readers the context). The nice things about these suggestions is that we don’t have to reinvent the wheel to implement them – as Kimberly mentions, we could see what historians and other historically-oriented disciplines already do to address questions of transparency.

I’d also add to that whether we should address ethical and legal issues pertaining to archives. These are less common issues in non human subject research, but they do come up. For instance, I use letters from private citizens to advocacy groups in my research. This raises questions about whether and how to protect their identities when it comes to citation, which I’ve dealt with on a case-by-case basis, depending a bit on the access restrictions of the archive. It would be nice to have clearer, more accepted guidance on these issues, though. The same goes for legal issues involving copyright and fair use when it comes to active citation – I think a lot of us who use archival work aren’t aware of potential copyright issues, or if we are we muddle through on an individual basis.

[Guest]

I completely agree with the suggestion that better footnoting would address the transparency problems. Having a standard set of things to identify in a footnote -- archive location, record group name, series, date, etc -- would greatly assist later researchers in identifying the sources used. Perhaps looking to historians for examples of how to do this well would help our discipline develop a consensus about what information needs to be included in footnotes.

I am also skeptical of the push to active citation, except where documents are already online. Otherwise, the time and resource burden poses a large barrier, especially to graduate students and junior faculty.

There is, however, one huge consequence of better footnoting that we need to address: word and space limits for journals. Including full archival citation in footnotes is both good research practice and essential for transparency. But unless journals are willing to provide additional space for qualitative researchers to include this citation information, we will (again) find this type of research at a disadvantage.

-- IR grad student

[Guest]

I am a big fan of Kimberly Morgan's meaty footnote strategy-- to provide specific information on archival locations, to elucidate the extant debate over a particular point. The degree to which a scholar can show an understanding of nuance and different interpretations, more than any embedded link, would help convince me of the validity of their scholarship.

Now don't get me wrong. I am not against embedded links where possible but I don't really see them as adding much value other than "proving" that the author did not make up the evidentiary basis of their claim. But to me, that's a strawman concern! How many people who know anything about qualitative work seriously believe that scholars are making up their evidence on a regular basis?

But I also agree with the previous poster-- that the increasingly tight word-limits at many "top" journals militate against this meaty footnote strategy. It's not such an issue at comparative journals like World Politics and CPS, or more qualitative journals like Politics & Society, SAPD, etc. But journals like the JOP etc (which my department considers a "top disciplinary journal", whereas the others I have mentioned are mere "field journals") have insanely low page/wordcount limits. In my mind, the JOP's 35-page (inclusive of all figures, tables and references) limit actually borders on anti-intellectual because it militates against proper citation strategies. Why are we as a discipline not talking about issues such as these?

Assistant professor at R1.

[Evgeny Finkel GWU]

I am absolutely in favor of meaty footnotes. I think that beyond time investment and data availability issues active citations create a serious problem, and that is the illusion that by a mouse click or two one gets access to everything that there is to be known. A meaty citation sends you first and foremost to an archive, and within the archive to a collection, a box, a folder, which explicitly means that the cited document does not exist in a vacuum; an active citation sends to one isolated piece of data and it is all to easy to forget about the broader context, especially if one have not worked in the archives before. Active citations encourage a hunt after smoking guns even when they don't exist (and we know how rarely one finds a perfect 'smoking gun' in archival documents).

[Mala Htun Univ of New Mexico]

A few thoughts:

--Meaty (or thorough) footnotes for qualitative work are ALREADY the equivalent of the new transparency and data access standards for quant work. Meaty footnotes point the reader to the data from which the ideas and conclusions in the article or book came, and enable the inquisitive reader to go back to those sources, the "data," and draw their own conclusions. That's exactly what making publicly machine readable datasets and code does--it enables inquisitive readers to go to the data, redo the analysis, and draw their own conclusions. it is important to note that DA-RT standards do not currently require that quant scholars provide access to RAW DATA, in the form of surveys, annotated reports or other documents that were subsequently coded, datasheets, etc. the raw data that goes into a stata/R/excel data file or spreadsheet is the functional equivalent of the transcripts and field notes that folks are objecting to sharing. if posting transcripts and field notes is required, then access to other forms of raw data should also be required.

--active citation is being pitched by Andy M. as a conservative "back to the future" mechanism to conform to DA-RT. compared to uploading and sharing field notes and transcripts, it is benign. but i agree with folks who say that a page or two of a transcript, taken out of context, is potentially meaningless. sharing such a transcript would also incur costs with anonymization, potentially, as well as raise the eyebrows of IRB and possibly involve acquiring informed consent to do so. this would involve the same host of problems as sharing full transcripts.

--i disagree with the idea of giving qualitative work longer word limits. articles based on quant data also have to make tough choices about what to include. it wouldn't be fair. what does make sense is for qual work to use "supplementary online materials" more aggressively. Andy M. calls this a "transparency appendix," but it could simply be a document with all the sources and ideas that the author wanted to go into the article but couldn't find a way to fit in.

mala

[Nancy Hirschmann The University of Pennsylvania]

Since active citation is one of the issues that potentially pertains to my own subfield of political theory, I wanted to note an interesting fact. With journals moving to "scientific notation" (in text reference to author, year, and page number) over the past 10 or 20 years, endnotes are increasingly discouraged--if it is important enough to mention, the philosophy goes, then it is important enough to put in the text. And even journals that permit endnotes that have a substantive content seek to limit them in length and number. This has had a significant effect on the political theory field, in the way that we are allowed to present our work to others. And it is done in the effort to copy science, as part of a historical progression that leads to DA-RT (severely compressing the genealogy here). So resistance to active citation and a move to bring back "meaty footnotes" could be very valuable in changing the conversation about what it is that we are up to in performing academic research and publishing in journals: that is, how we seek to communicate with the wider academic audience, our readers. I'm all for "bringing the footnote back" as an argument against active citation.

[Kimberly Morgan George Washington University]

Thanks everyone for your responses. I'm happy to see I'm not the only one out there who likes chunky footnotes. I think Nancy's point is an important one -- that to the extent journals move away from footnoting to in-text citation (while also often discouraging footnotes/endnotes), and impose 8000-word limits (as R1 assistant professor and IR grad student note), it becomes very difficult for qualitative scholars to say much of significance while also meeting various transparency goals. My sense is that if we want to move qualitative work toward greater transparency, we need to push back against this. I can see Mala's point about how it might be seen as unfair to give qualitative work longer word limits, but if we're serious about enabling qualitative work to flourish in the discipline, we need to take seriously the idea that qualitative and quantitative work may need different things from journals. I fear some journals have moved toward a quantitative publication model without thinking much about it, the 8,000-word limit and in-text citation being cases in point.

[Marcus Kreuzer Villanova University]

Given that we are among footnote aficionados here, just a quick plug for Anthony's Grafton marvelous history "The Footnote: a Curious History." He traces the history of the footnote back to the 17th century's scientific revolution and Rene Descartes disparagement of historical knowledge as lacking scientific standing. Historians responded by starting to use footnotes to make their research more transparent, replicable, and hence scientific. Some the later chapters also discuss the changing uses of footnotes in history and can be easily read as the qualitative equivalent of null hypothesis testing. Grafton points out that a common function of footnotes involves discussing alternative interpretations than the ones offered in the main narrative and thus addressing counter-evidence and counter-evidence in a manner somewhat relate to the function the null hypothesis serves in large n research.
But he also points out that the footnote only can serve this function effectively on if it is not constrained by word limits and if cited at the bottom of the pages rather than being banished to the back of an article or book.
So, if journals want to be serious about transparency give us back the chunky footnote and make quantitative scholars document their research judgments in a chunkier fashion as well. A good corrective to their epistemological anorexia would be to insist on having a page number !!

[robert mickey university of michigan--ann arbor]

I think active citation (AC) is a good norm to follow. However, the thing that AC does *NOT* help with--and this is of course related to many mainstream scholars' distrust of/derision toward qual work--is that if a document from the archives is the numerator, I can't show those scholars the denominator. We usually can't describe a precise "data-generating process" by which the archives were produced. So when I write that an archive was surprisingly absent of discussions about issue X, as I did a few times in my book, I can't provide any evidence besides "trust me." I'd be curious what's the share of archives that have been digitized. The costs of doing so are falling dramatically, but so are librarians' budgets.

[Nikhar Gaikwad Yale University]

I would like to expand a little bit more on Mala Htun's third point about journal word limits and supplementary appendices.

First of all, I agree that word/page limits in journals such as AJPS and JOP are simply too low to permit meaty footnotes and other forms of extensive citations that provide full contextual background to source documents within published versions of articles. Requiring qualitative scholars to provide such detailed forms of references within manuscripts, without correspondingly increasing journal word limits, is unfair; it will effectively shut many forms of qualitative research out of the top disciplinary journals in Political Science. But asking for different word limits for qualitative and quantitative scholarship is tricky. Who will decide whether a manuscript is qualitative or quantitative, and what about the many forms of research today that employ mixed-methods approaches? As Mala Htun points out, different word limits might not be fair to quantitative scholars, who also have to make tough decisions about the level of detail to include when describing data preparation and analysis. And from a practical perspective, due to pricing and printing constraints in the publishing industry, journals are already squeezed when it comes to page limits. Is it realistic to expect that journals will agree to substantially increase word limits for qualitative scholars? I think that we will need clear answers to these questions before being able to fully evaluate the costs and benefits of meaty footnotes and other forms of detailed citations within published versions of articles.

One way out of this quagmire is to have qualitative research rely more on supplementary/online appendices. Since these appendices do not have word limits, scholars will have a fair amount of latitude to explain their sources and provide detailed contextual background to each citation. If we go this route, we might not even have to decide between meaty footnotes and active citation methods---why not have both? That is, the standard could be to have brief citations in the published version of an article. This will allow qualitative scholars to meet journal word limits in a reasonable manner. Then, the supplementary appendix could include a "detailed citation" version of the same published article. Here, authors can include meaty footnotes that provide the necessary context to each source. But each footnote can also be linked to the page(s) from the source documents where specific quotes or references are drawn from. This way, other researchers who wish to replicate the findings of the paper can quickly find the immediate backup for each citation using the active citation approach. But if the abridged backup is insufficient, they will have access to all of the information in the meaty footnotes that will allow them to return to the archives (or other repositories where source materials are located) and verify the manuscript's claims. These supplementary appendices can be made available to reviewers (in an anonymous fashion) during the review process, if we want to ensure that appropriate vetting takes place prior to publication.

My point is that we might not necessarily have to make a choice between active citations and meaty footnotes. I think we can have both if we rely on supplementary appendices. (Of course, whether active citations place undue cost burdens on qualitative scholars is another matter worthy of its own debate.)