Qualitative Transparency Deliberations

[Veronica Herrera University of Connecticut]

What are the benefits of increasing transparency for how we obtain our text based and non text based sources, and how we use these sources to make analytical, descriptive or causal claims?

What are the costs of increasing transparency for how we obtain our text and non text based sources, and how we use our these sources to make analytical, descriptive or causal claims? Do you have suggestions for practices that can help mitigate the costs of increasing data transparency for text-based and non-text based sources?

[Nikhar Gaikwad Yale University]

I see several important benefits of increasing transparency for both text-based and non-text-based sources. To start the conversation, one major advantage of increased transparency is that it can help provide consumers of research with contextual information that can, in turn, help them evaluate or consider sources of bias in the data used to buttress evidentiary claims.

As a concrete example, archives typically tend to retain particular types and sources of data that, intentionally or not, might reflect only a partial rendering of the historical record. Relying only on sources that "survived," without providing readers with information about the types of sources that might not have been retained, could thus introduce bias and lead to misleading interpretations of historical events.

Even when presenting evidence from text-based sources that are available in archives, researchers can generate more confidence in their claims by discussing the research designs that guided their data analyses in the first place. For example, manuscripts oftentimes present selected quotes in order to support particular interpretations of the data. But without additional contextual information, it can be difficult for readers to properly evaluate the arguments in question. What was the set of documents that was reviewed over the course of the research? Were particular sources omitted from the analyses? Are the quotes selected for inclusion in the manuscript representative of the full set of documents? Did the author set out only to verify a hypothesis or did the author also search for evidence to refute the hypothesis? Were alternative hypotheses tested with the data?

By making available answers to questions such as these, researchers can strengthen their descriptive or inferential claims, and address selection concerns and concerns related to bias that discerning readers might have.

[Sam Handlin University of Utah]

I have written one "TRAX" (transparency appendix) for a paper. In this document, for every primary source (all textual) that I referenced, I supplied a full bibliographic reference, a more extended quotation (usually about 150 words) from the passage that was supporting the relevant inference drawn in the paper, and a short paragraph explicitly laying out why this piece of evidence supported that inference. This was a very useful exercise in academic self-discipline, which forced me to carefully consider whether sources were actually supporting my claim (in a broad sense, not just cherry picking quotes) and to explicitly lay out in detail the connection between evidence and inference.

By their nature, these transparency discussions put great emphasis on the idea of scholarly interchange - making sure that others can fully evaluate your claims, that they can find your sources, that they could potentially replicate your work, etc. I don't want to minimize the importance of that kind of academic back-and-forth, which is clearly at the heart of these initiatives. But I think the biggest benefit of adopting transparency practices is simply that it makes your own research stronger. We are all tempted to cut corners at times and this is a way to discipline yourself against those inclinations.

My biggest concern is that there seem to be very unclear instrumental benefits to transparency at this point. Is it advantageous to submit a TRAX like the one described above as an appendix to a journal article (assuming for the moment that the journal is not requiring that)? Some reviewers may see this a laudable exercise in transparency and view a TRAX as a point in the submission's favor. But others may see this as a bunch of new material to sort through and nitpick.* Given that it often only takes one reviewer to sink a paper, it is not completely obvious to me that offering up an additional 30-40 pages of extended quotations from textual sources and commentary on my interpretation and use of them is actually going to increase the likelihood of getting a paper accepted. Writing a thorough and careful TRAX involves a lot of work for relatively uncertain rewards. Particularly for junior faculty, I don't know if it makes that much sense given current incentive structures.

*In my opinion, a TRAX is not really like a quantitative appendix in this regard, which (de facto at least) tends to present a bunch of information and additional statistical results that are carefully selected by the author to be relatively bullet proof. Presenting big sections of text from your sources in a TRAX, in contrast, will always open you up to potential criticism from others about the interpretation of that text. That kind of criticism and exchange is healthy from an academic standpoint. But I'm not fully convinced that it is advantageous from the instrumental standpoint of maximizing your odds of manuscript acceptance.

[Veronica Herrera University of Connecticut]

Sam, thanks so much for these insights. It sounds like the benefits of a transparency appendix from your perspective is self discipline and a more rigorous scholarly product, while the downside is that it may provide reviewers with another reason to nix the paper if they disagree with the interpretation of the material or are irritated to have so much more to review.

Are there ways to get around these potential downsides? Should transparency appendices be mandatory for manuscript reviewers to read through, or can that review process be at the reviewer's discretion? To make transparency appendices more manageable for both the author and the reviewer, are there some key things that need to be in there (if so what are they?) and others that could be optional in order to make it more concise? Are there alternatives to the full transparency appendix that still advances transparency goals beyond the footnote?

I have written one "TRAX" (transparency appendix) for a paper. In this document, for every primary source (all textual) that I referenced, I supplied a full bibliographic reference, a more extended quotation (usually about 150 words) from the passage that was supporting the relevant inference drawn in the paper, and a short paragraph explicitly laying out why this piece of evidence supported that inference. This was a very useful exercise in academic self-discipline, which forced me to carefully consider whether sources were actually supporting my claim (in a broad sense, not just cherry picking quotes) and to explicitly lay out in detail the connection between evidence and inference.

By their nature, these transparency discussions put great emphasis on the idea of scholarly interchange - making sure that others can fully evaluate your claims, that they can find your sources, that they could potentially replicate your work, etc. I don't want to minimize the importance of that kind of academic back-and-forth, which is clearly at the heart of these initiatives. But I think the biggest benefit of adopting transparency practices is simply that it makes your own research stronger. We are all tempted to cut corners at times and this is a way to discipline yourself against those inclinations.

My biggest concern is that there seem to be very unclear instrumental benefits to transparency at this point. Is it advantageous to submit a TRAX like the one described above as an appendix to a journal article (assuming for the moment that the journal is not requiring that)? Some reviewers may see this a laudable exercise in transparency and view a TRAX as a point in the submission's favor. But others may see this as a bunch of new material to sort through and nitpick.* Given that it often only takes one reviewer to sink a paper, it is not completely obvious to me that offering up an additional 30-40 pages of extended quotations from textual sources and commentary on my interpretation and use of them is actually going to increase the likelihood of getting a paper accepted. Writing a thorough and careful TRAX involves a lot of work for relatively uncertain rewards. Particularly for junior faculty, I don't know if it makes that much sense given current incentive structures.

*In my opinion, a TRAX is not really like a quantitative appendix in this regard, which (de facto at least) tends to present a bunch of information and additional statistical results that are carefully selected by the author to be relatively bullet proof. Presenting big sections of text from your sources in a TRAX, in contrast, will always open you up to potential criticism from others about the interpretation of that text. That kind of criticism and exchange is healthy from an academic standpoint. But I'm not fully convinced that it is advantageous from the instrumental standpoint of maximizing your odds of manuscript acceptance.

[Nikhar Gaikwad Yale University]

To get the conversation about "costs" of transparency started, I would like to point out one replication drawback potentially faced by scholars relying on historical text-based sources from archives.

Many archives have strict limits on the number of documents that any one scholar can photocopy or reproduce. In other words, one might be able to obtain the "backup" for particular pieces of evidence, but might not be able to access the full set of documents that was analyzed over the course of one's research in the archives. Moreover, certain archives do not permit any form of reproduction of source materials.

In such instances, it can be very difficult, if not impossible, to hold scholarly work relying on historical text-based sources to the same replication standards as large-n quantitative research.

[Ingo Rohlfing Cologne Center for Comparative Politics, Universität zu Köln]

This is an interesting point I did not know before. But I do not see an inherent problem here. If an archive imposes such constraints, they are publicly documented somehow and should be reported in the publication. Nobody can expect one to be more transparent than an archive allows one to be.

To get the conversation about "costs" of transparency started, I would like to point out one replication drawback potentially faced by scholars relying on historical text-based sources from archives.

Many archives have strict limits on the number of documents that any one scholar can photocopy or reproduce. In other words, one might be able to obtain the "backup" for particular pieces of evidence, but might not be able to access the full set of documents that was analyzed over the course of one's research in the archives. Moreover, certain archives do not permit any form of reproduction of source materials.

In such instances, it can be very difficult, if not impossible, to hold scholarly work relying on historical text-based sources to the same replication standards as large-n quantitative research.

[Ingo Rohlfing Cologne Center for Comparative Politics, Universität zu Köln]

This is a very interesting post and it seems like you put a lot of effort into your TRAX. I understand there are concerns about how reviewers respond to a TRAX, but I do not share them. If a reviewer wants to bring you down, the reviewer will find a way. If you have a TRAX, it might be something in it; if you do not have a TRAX, you are criticized for being intransparent or because of something else.

If you have a TRAX, I would always show it because it demonstrates how much effort you put into the management of sources and interpretation of evidence. All else equal, a publication with a TRAX is always better than one without, if only because it disciplines your own reasoning and use of sources.

I have written one "TRAX" (transparency appendix) for a paper. In this document, for every primary source (all textual) that I referenced, I supplied a full bibliographic reference, a more extended quotation (usually about 150 words) from the passage that was supporting the relevant inference drawn in the paper, and a short paragraph explicitly laying out why this piece of evidence supported that inference. This was a very useful exercise in academic self-discipline, which forced me to carefully consider whether sources were actually supporting my claim (in a broad sense, not just cherry picking quotes) and to explicitly lay out in detail the connection between evidence and inference.

By their nature, these transparency discussions put great emphasis on the idea of scholarly interchange - making sure that others can fully evaluate your claims, that they can find your sources, that they could potentially replicate your work, etc. I don't want to minimize the importance of that kind of academic back-and-forth, which is clearly at the heart of these initiatives. But I think the biggest benefit of adopting transparency practices is simply that it makes your own research stronger. We are all tempted to cut corners at times and this is a way to discipline yourself against those inclinations.

My biggest concern is that there seem to be very unclear instrumental benefits to transparency at this point. Is it advantageous to submit a TRAX like the one described above as an appendix to a journal article (assuming for the moment that the journal is not requiring that)? Some reviewers may see this a laudable exercise in transparency and view a TRAX as a point in the submission's favor. But others may see this as a bunch of new material to sort through and nitpick.* Given that it often only takes one reviewer to sink a paper, it is not completely obvious to me that offering up an additional 30-40 pages of extended quotations from textual sources and commentary on my interpretation and use of them is actually going to increase the likelihood of getting a paper accepted. Writing a thorough and careful TRAX involves a lot of work for relatively uncertain rewards. Particularly for junior faculty, I don't know if it makes that much sense given current incentive structures.

*In my opinion, a TRAX is not really like a quantitative appendix in this regard, which (de facto at least) tends to present a bunch of information and additional statistical results that are carefully selected by the author to be relatively bullet proof. Presenting big sections of text from your sources in a TRAX, in contrast, will always open you up to potential criticism from others about the interpretation of that text. That kind of criticism and exchange is healthy from an academic standpoint. But I'm not fully convinced that it is advantageous from the instrumental standpoint of maximizing your odds of manuscript acceptance.

[Jane Mansbridge Harvard Kennedy School]

My worry on this matter is simply a cost-benefit one. The work of producing a TRAX is undoubtedly good for the researcher and for the reader. But how good, compared to starting work on another important subject? We have limited lives and very limited research time. I haven't seen much discussion of the problem to which TRAX is a solution. I got my MA at Harvard in History and never heard discussions of problems to which TRAX appendices would be the solution. This is not to say that such problems might exist, and perhaps be very important in some cases. But before adopting TRAX as best practice, it would be great to hear of several examples of bad problems to which TRAX would be the answer.

[Nikhar Gaikwad Yale University]

Thank you for your perspectives, Sam, Ingo, and Jane.

At least two distinct tradeoffs appear to emerge in the discussion of the costs and benefits of a TRAX. One pertains to the journal publication and review process. A TRAX might increase the strength of a manuscript and give reviewers (and authors and readers) greater confidence in its use of evidence. At the same time, it might hold qualitative research up to higher standards than other types of research and give reviewers additional reasons to reject a manuscript; as Sam mentions, "presenting big sections of text from your sources in a TRAX...will always open you up to potential criticism from others about the interpretation of that text."

A second tradeoff pertains to the time and effort it takes to produce a TRAX. Jane points out that even if a publication with a TRAX is superior to a publication without a TRAX, scholars might be better off directing the time and effort potentially spent in producing a TRAX to other research endeavors.

Are there solutions that can mitigate each of these tradeoffs, either independently or in conjunction? For example, what if a TRAX did not contain extended sections of text, but simply quotations to support (or interrogate) the author's claims? Would page numbers to secondary source references (as suggested in a related discussion thread on this board) suffice? It will also be helpful to hear, as Jane suggested, "several examples of bad problems to which TRAX would be the answer."

[Marcus Kreuzer Villanova University]

Two possible solutions come to mind:
1) The author submitting a TRAX and increasing his/her transparency should be rewarded by being the right to respond to any reviewer's critiques drawn from the TRAX. This might involve restructuring the review process. Maybe reviewers would be invited to comment on transparency related issues in a discreet section. If this section raises significant transparency concerns, then the editor should give the author an opportunity to respond to those specific transparency considerations.
2) The editor might also invite the other reviewers to weigh in when reviewers raise some substantial transparency considerations.
These two relatively minor modifications would have two benefits. First, it would incentivize authors to build and submit TRAX. The benefit would no longer just the credibility gain that comes with a TRAX but also with the protection against unsubstantiated criticisms that could kill a manuscript. Second, it also would dis-incentivize the reviewer of taking cheap shots at a manuscript.

Thank you for your perspectives, Sam, Ingo, and Jane.

At least two distinct tradeoffs appear to emerge in the discussion of the costs and benefits of a TRAX. One pertains to the journal publication and review process. A TRAX might increase the strength of a manuscript and give reviewers (and authors and readers) greater confidence in its use of evidence. At the same time, it might hold qualitative research up to higher standards than other types of research and give reviewers additional reasons to reject a manuscript; as Sam mentions, "presenting big sections of text from your sources in a TRAX...will always open you up to potential criticism from others about the interpretation of that text."

[Guest]

Thinking about the issue of transparency in text-based research brought to mind a 1974 essay by the historian Jack Hexter, "The Historical Method of Christopher Hill." I believe that this essay is best known for popularizing the distinction between "lumpers" and "splitters," but Hexter uses this distinction in service of a broader argument about the pitfalls of historical research. Hexter does so in the context of a book review of the eminent historian Christopher Hill, whom Hexter acknowledges to have unparalleled command of the archival and historical materials in his area of expertise (16th–17th-century England). But, Hexter argues, it is Hill's very erudition that permits him to abuse or misinterpret historical evidence in service of the argument he wishes to make. Given the vastness of the textual material at this command (coupled with his tendency to "lump", or elide nuances and complications), Hill possesses an almost limitless capacity to find evidence for whatever argument he wishes to make. Hester calls this "source-mining," which intentionally or not strongly evokes the concern with "data-mining" or "p-hacking" now so prevalent among quantitative historians. Hexter notes that other scholars, unable to match Hill's limitless knowledge of the historical evidence, have little hope of checking Hill's source-mining. The best they can do is to engage in a sort of qualitative "replication." Hexter proceeds to perform such a replication on a few paragraphs of Hill's new book, finding that Hill's conclusions are often based on simplistic or misleading, though not blatantly false, interpretations of the textual evidence. "Given his erudition and his predisposition to lumping," Hexter writes, Hill "is bound to find evidence that seems to support his case" and thus "can be sure of arriving at any conclusion he aims at." In other words, Hexter concludes, it is the most knowledgeable and imaginative scholars who may be at greatest risk of this sort of misrepresentation of the historical evidence. While I agree that this may be true, I interpret Hexter's critique less as an indictment of Hill personally than as a commentary on the pitfalls of historical and text-based research. Even historical researchers of ordinary erudition are constantly in the position of reviewing archival material that few others have ever examined and even fewer could or would check to ensure that their interpretation is correct. The same goes for interview-based evidence. It is possible that randomly sampled "replication" of the sort performed by Hexter may help with this issue. But in the end I tend to agree with Hexter's conclusion that "each historian lives under an especially heavy obligation to police himself."

[Guest]

[quote="jane mansbridge"]My worry on this matter is simply a cost-benefit one. The work of producing a TRAX is undoubtedly good for the researcher and for the reader. But how good, compared to starting work on another important subject? We have limited lives and very limited research time. I haven't seen much discussion of the problem to which TRAX is a solution. I got my MA at Harvard in History and never heard discussions of problems to which TRAX appendices would be the solution. This is not to say that such problems might exist, and perhaps be very important in some cases. But before adopting TRAX as best practice, it would be great to hear of several examples of bad problems to which TRAX would be the answer.[/quote]

This is an important point. The problem that transparency is supposed to be addressing isn't at all clear, and yet the costs of increasing transparency - especially for qualitative scholars - are potentially quite high. It seems like we're spending a lot of time and thought discussing - and potentially implementing - a costly solution to a relatively non-existent problem.

[Guest]

I think a bigger question that remains unaddressed is whether or not it is the responsibility of the researcher using archival sources to make all of the sources available. Leaving aside limitations of photocopies or scans, to require a researcher to post images of all of her documents places a huge burden on the researcher -- particularly if they are a junior scholar or graduate student. Archival research is already time-intensive; requiring us to upload images of our text or image documents would be yet another burden placed on the qualitative researcher. I am wary that defaulting to the limitations on photocopies imposed by archives implicitly assumes that it is always good to require us to upload document images.

Instead, archival research would benefit from systematic and consistent footnoting practices that identify all of the pieces of information another researcher would need to find the documents him or herself. But assuming that it is the researcher's responsibility to collect, process, and upload documents will make archival research even more prohibitively costly than it already is.

[quote="ingorohlfing"]This is an interesting point I did not know before. But I do not see an inherent problem here. If an archive imposes such constraints, they are publicly documented somehow and should be reported in the publication. Nobody can expect one to be more transparent than an archive allows one to be.

[quote="Nikhar Gaikwad"]To get the conversation about "costs" of transparency started, I would like to point out one replication drawback potentially faced by scholars relying on historical text-based sources from archives.

Many archives have strict limits on the number of documents that any one scholar can photocopy or reproduce. In other words, one might be able to obtain the "backup" for particular pieces of evidence, but might not be able to access the full set of documents that was analyzed over the course of one's research in the archives. Moreover, certain archives do not permit any form of reproduction of source materials.

In such instances, it can be very difficult, if not impossible, to hold scholarly work relying on historical text-based sources to the same replication standards as large-n quantitative research.[/quote][/quote]

[Veronica Herrera University of Connecticut]

These are great points, pushing us to think about the idea of undue burden on researchers....

I think a bigger question that remains unaddressed is whether or not it is the responsibility of the researcher using archival sources to make all of the sources available. Leaving aside limitations of photocopies or scans, to require a researcher to post images of all of her documents places a huge burden on the researcher -- particularly if they are a junior scholar or graduate student. Archival research is already time-intensive; requiring us to upload images of our text or image documents would be yet another burden placed on the qualitative researcher. I am wary that defaulting to the limitations on photocopies imposed by archives implicitly assumes that it is always good to require us to upload document images.

Instead, archival research would benefit from systematic and consistent footnoting practices that identify all of the pieces of information another researcher would need to find the documents him or herself. But assuming that it is the researcher's responsibility to collect, process, and upload documents will make archival research even more prohibitively costly than it already is.

This is an interesting point I did not know before. But I do not see an inherent problem here. If an archive imposes such constraints, they are publicly documented somehow and should be reported in the publication. Nobody can expect one to be more transparent than an archive allows one to be.

To get the conversation about "costs" of transparency started, I would like to point out one replication drawback potentially faced by scholars relying on historical text-based sources from archives.

Many archives have strict limits on the number of documents that any one scholar can photocopy or reproduce. In other words, one might be able to obtain the "backup" for particular pieces of evidence, but might not be able to access the full set of documents that was analyzed over the course of one's research in the archives. Moreover, certain archives do not permit any form of reproduction of source materials.

In such instances, it can be very difficult, if not impossible, to hold scholarly work relying on historical text-based sources to the same replication standards as large-n quantitative research.

[Shamira Gelbman Wabash College]

This is an interesting point I did not know before. But I do not see an inherent problem here. If an archive imposes such constraints, they are publicly documented somehow

[/quote]

Not necessarily. Some small archives might only have that constraint documented on a laminated handout or contract they have you read and sign when you show up to do your research. Some 'archives' are a proprietary box of papers in a private individual or organization's storage closet, and they may not have their reproduction policies written out at all.

[Shamira Gelbman Wabash College]

This is an interesting point I did not know before. But I do not see an inherent problem here. If an archive imposes such constraints, they are publicly documented somehow

Not necessarily. Some small archives might only have that constraint documented on a laminated handout or contract they have you read and sign when you show up to do your research. Some 'archives' are a proprietary box of papers in a private individual or organization's storage closet, and they may not have their reproduction policies written out at all.

I should add, some archives also change their policies, sometimes very suddenly and without maintaining record of the old rules -- what their publicly documented policy on reproduction says now might not be the one that was in place when the researcher was there. Particularly for those whose jobs (e.g. with high teaching/service loads) make the research process even slower than it already is for archival research, publication requirements that look to archives' current publicly documented policies can be inordinately burdensome.

[Ingo Rohlfing Cologne Center for Comparative Politics, Universität zu Köln]

My worry on this matter is simply a cost-benefit one. The work of producing a TRAX is undoubtedly good for the researcher and for the reader. But how good, compared to starting work on another important subject? We have limited lives and very limited research time. I haven't seen much discussion of the problem to which TRAX is a solution. I got my MA at Harvard in History and never heard discussions of problems to which TRAX appendices would be the solution. This is not to say that such problems might exist, and perhaps be very important in some cases. But before adopting TRAX as best practice, it would be great to hear of several examples of bad problems to which TRAX would be the answer.

To me, the problems are obvious. The famous piece referenced in this context is the reanalysis of one part of Moravcsik's book by Lieshout ( Lieshout, R. H.; Segers, M. L. L. & Vleuten, A. M. v. d. De Gaulle, Moravcsik, and The Choice for Europe: Soft Sources, Weak Evidence Journal of Cold War Studies, 2004, 6, 89-139). It turns out there are a lot of problems with the way sources are referenced and interpreted by Moravcsik. Lieshout et al. could do the reanalysis based on the book without a TRAX, but a TRAX would have been helpful, I assume. Moreover, some of the mistakes might have been avoided if Moravcsik had compiled a TRAX because it disciplines your use of sources. This article is mostly cited in this context because people rarely go back to the sources of a publication and do a reanalysis, that is, it might seem there is no bigger problem at stake. But I do not see any reason to believe that Moravcsik is an outlier and most of the other studies reference properly and interpret evidence correctly.

[Ingo Rohlfing Cologne Center for Comparative Politics, Universität zu Köln]

But in the end I tend to agree with Hexter's conclusion that "each historian lives under an especially heavy obligation to police himself."

I agree that a researcher should police herself. But this does not imply that others are relieved from policing researchers; "policing researchers" actually is not a good term because it is about a) research and not researchers, b) increasing confidence in the conclusions at hand instead of being a "methods police". In this spirit, I also believe it is the responsibility of a researcher to make it as easy as possible for others, within reasonable constraints, to recapitulate how one arrived at one's conclusions. This is now a widely held opinion in quantitative research and I do not see a difference to qualitative research in this respect.

[Ingo Rohlfing Cologne Center for Comparative Politics, Universität zu Köln]

This is an important point. The problem that transparency is supposed to be addressing isn't at all clear, and yet the costs of increasing transparency - especially for qualitative scholars - are potentially quite high. It seems like we're spending a lot of time and thought discussing - and potentially implementing - a costly solution to a relatively non-existent problem.

In the first place, one should consider a TRAX (or any equivalent) as beneficial for oneself because it disciplines your own research. After half a year, a TRAX easily allows oneself to understand what sources one used and what one inferred from them. It is the same with quantitative research. You might try to rush through some analysis, but after two months you find it very difficult to reproduce your own results if you do not have generated an annotated script. This is less work then writing a TRAX, but a good annotated script also requires some effort. Producing and updating a TRAX during the actual research process is also less costly, at least psychologically, because the work spread is spread over a longer period.

I understand that there is a natural preference for doing new research instead of working on a TRAX. This issue would diminish if one makes the TRAX part of a research process instead of considering it an addendum to be done after publication of an article. Besides, I strongly believe we should turn away from the belief that new research is better or more productive than spending more effort on the research we do. Of course, tenure committees and funding agencies are likely to value an additional study more than one with an excellent TRAX (in times of DA-RT, this might change, though). But we should focus more on doing the research we do more systematically instead of jumping from one study to the next.

[Guest]

In addition to conducting qualitative research, I also conduct quantitative research. As Ingo suggests, I think some consideration of the rise of the lengthy quantitative appendix is useful when thinking about what our field would look like if we also encouraged the rise of the lengthy qualitative appendix.

Just 5-10 years ago, when I submitted quantitative papers to journals, I did not submit lengthy appendices -- either I did not submit an appendix at all, or I submitted an appendix of only 1-2 pages, which typically described some key facet of variable coding or the sample. Today, when I submit quantitative papers to journals, I always submit an appendix, and this appendix will be 20-40 pages long in the end and often longer than the paper itself. In addition to extensive information about data collection, sampling, measurement, and background on my case(s), my appendix will also contain at least 10-20 additional tables and figures. Any robustness check discussed in the text appears in full in the appendix. This type of appendix is in no way formally required by journals, but my impression is that reviewers increasingly expect to see it from authors. I want to send a signal that I am a careful scholar, so I make a lengthy appendix.

Making this kind of appendix is not trivial. Maybe/probably it is a lot easier than making a TRAX (I've never made one), but believe me, it is NOT easy. It actually takes a lot of work -- which is why it sends a signal about the quality of my paper. And like Sam who commented above, it has been a "very useful exercise in academic self-discipline" for me. I have to essentially treat every robustness check as carefully as I would my main analysis. And since my quantitative research almost always involves original data collection, I have to be much more careful to document every step of the data collection process so that I can report it months/years later in the appendix. I feel more confident in my research as a result. That being said, it takes a lot of time -- and it's not as if people are actually reading my appendix... But I want my research to get published! So I do it.

I think that this signaling dynamic suggests that -- regardless of whether DA-RT or JETS creates formal incentives for qualitative researchers to create a lengthy appendix -- we might start seeing them because of the signaling dynamic I outline above. To again speak from my personal experience: I considered making a TRAX for my most recent qualitative paper and decided against it. I figured that it would be a useful exercise and probably improve the research on the margins. But at the same time, I figured that it would take a lot of time and that no one expects to see one yet. (I've never reviewed a qualitative article that made use of active citation or TRAX.) So I thought the trade-off in terms of my time wasn't worth it.

But I wonder if it will stay that way, regardless of the outcomes of these deliberations. My experience with quantitative research leads me to be cynical/pessimistic. Someone is going to submit an article with an accompanying TRAX, and then someone is going to review it and think, "huh, this is impressive." (You can see how this dynamic might start to emerge from some of the earlier comments on this thread -- even if we don't demand a TRAX, at least some people are going to think it looks good.) And then more people will start sending in TRAX to send a signal about the quality of the research, and it might become the norm or expectation. So I'm not really sure what "we" as a profession can do to stop people from upping the ante in terms of the signal that they're trying to send in terms of their research quality. If we're concerned about inequalities or the time lost to us as qualitative researchers through having to create TRAX (or whatever else), then perhaps part of the solution should be to create systems to make them less costly, especially to under-resourced researchers. Or perhaps journals should create upper limits for appendices shared with reviewers.

[Jacques Hymans (University of Southern California)]

Yes very true. I was given access to a private archive because the individual trusted me. He did not want anyone else to see the papers, and in fact he did not want anyone else to know that he had the papers, because he did not trust that some unknown stranger would treat the information in those papers with the care and objectivity that I would do.
Jacques Hymans
University of Southern California

[Guest]

This is Taylor Boas from Boston University. I'm posting as "guest" since I don't yet have a login here. I think the Dec. 9 anonymous poster (two up from this one) makes excellent points about appendices as signaling devices, whether for quant or qual research. Sam suggests that appendices in quant papers are generally bulletproof and strengthen the argument rather than exposing the author to additional criticism, but I don't think that's necessarily the case. I have had reviewers hone in one things in quantitative appendices as grounds for criticism and, presumably, a rejection vote. There are things that are expected in quant appendices that are the parallel to Sam's example of extra context from which a quote was extracted. For example, it is common now to present results for one's main independent variable(s) graphically (e.g., a dot-plot with confidence intervals), while providing a full table of coefficient estimates and standard errors, including for control variables, in the Appendix. The additional "context" of results for the control variables is easily something that a reviewer could seize upon and criticize. For both quant and qual research, the question of including an appendix (and what goes in it) is ultimately a calculation that authors will need to make--the signal of careful research that is conveyed by an appendix, weighted against the potential vulnerability of making additional detail public (times the probability that reviewers will actually delve into an appendix). And the extent to which appendices are emerging as a norm are a crucial part of this calculation. Yet even at present, before they have emerged as a norm, I think they send an important signal of seriousness, careful analysis, and transparency. I feel that way, for example, about Tasha Fairfield's process-tracing appendix in her World Development article. And in Tasha's case this helped springboard a new research agenda on Bayesian process-tracing. So being at the leading edge of a transparency practice can have payoffs down the road in unanticipated ways.

[Amy Poteete Concordia University]

Sam describes a TRAX as involving "an additional 30-40 pages of extended quotations from textual sources and commentary on my interpretation and use of them." The guest who discusses quantitative appendices as signalling devices says that they are "20-40 pages long in the end and often longer than the paper itself." I am reminded also of Tasha Fairfield's contribution to the 2015 to the QMMR newsletter on her experiences with documenting evidence for process-tracing, which highlighted again the effort taken to compile such documentation, its length, etc (https://www.maxwell.syr.edu/uploadedFil ... 2013_1.pdf).

I can easily believe that a TRAX or quantitative appendix will indeed be 20 - 40 pages and often longer than the article itself, especially given the restrictive word limiits journals place on manuscripts. But, for me, this really raises serious questions about journal policies and what it is that we are trying to do. There is the time issue, mentioned by Jane. One could say that good research requires time and that's just the way it is. To some extent, this might be seen as comparable to code books and logs of analyses, whether for quantitative or qualitative data and analysis. We should maintain good documentation, for our own records, regardless of whether we intend to share them or anybody asks for them.

But then, if there is a possibility that some journals will begin REQUIRING such documentation, I think we should ask ourselves some serious questions. First, does it make sense to require supplemental material that is as long or longer than the material to be published? This is akin to doubling the time and effort required for each publication. Would the quality of published work actually improve? If so, maybe it is worth it. It would certainly slow the time to publication, potentially significantly, and that is an important cost. Slower time to publication implies fewer publications when on the job market or up for review, tenure, or promotion. Will expectations shift to account for changing practices? Maybe, gradually, but that seems little comfort for tenure-track scholars. Equally important, slower time to publication affects the possibilities for more public scholarship, for engaging in and influencing public debates through our publications. There are, of course, other outlets (e.g., blogs) that allow more rapid publication. These outlets, by their very nature, require less documentation that formal peer-reviewed publications. That might be okay when one can refer to the more complete analysis. Still, we should ask, what are the costs of an increased lag between publication via blogs and publication in peer-reviewed outlets? Beyond public scholarship, slower time to publication would seem to slow scholarly exchange and thus our ability to learn from one another.

Why are we relegating more detailed evidence and documentation to appendices that are either only sent to reviewers without being published or are published as online appendices? I have long been frustrated with word limits that have limited my ability to provide references that go beyond attributing material to signalling the breadth of consensus across sources, contextual footnotes that speak to ambiguities in the data, and more expansive discussion of process-tracing. Another thread debated the idea of having an appendix of "meaty" references, which I understand as moving what would otherwise be end notes into an entirely separate document. It is difficult for me to see how this would work well. As I understand it, it would either mean leaving the main document without references (because they are all in this separate document) or it would mean having two versions of the references, with the appendix version more complete but less closely linked to the actual use of the material. To me, the better solution would be to loosen the word limits with the understanding that a greater proportion of the word counts should be devoted to documentation than is currently the case.

[Nikhar Gaikwad Yale University]

Thank you for sharing these very helpful perspectives, Ingo, Shamira, Jacques, and Guest.

This is an important point. The problem that transparency is supposed to be addressing isn't at all clear, and yet the costs of increasing transparency - especially for qualitative scholars - are potentially quite high. It seems like we're spending a lot of time and thought discussing - and potentially implementing - a costly solution to a relatively non-existent problem.

In the first place, one should consider a TRAX (or any equivalent) as beneficial for oneself because it disciplines your own research. After half a year, a TRAX easily allows oneself to understand what sources one used and what one inferred from them. It is the same with quantitative research. You might try to rush through some analysis, but after two months you find it very difficult to reproduce your own results if you do not have generated an annotated script. This is less work then writing a TRAX, but a good annotated script also requires some effort. Producing and updating a TRAX during the actual research process is also less costly, at least psychologically, because the work spread is spread over a longer period.

I understand that there is a natural preference for doing new research instead of working on a TRAX. This issue would diminish if one makes the TRAX part of a research process instead of considering it an addendum to be done after publication of an article. Besides, I strongly believe we should turn away from the belief that new research is better or more productive than spending more effort on the research we do. Of course, tenure committees and funding agencies are likely to value an additional study more than one with an excellent TRAX (in times of DA-RT, this might change, though). But we should focus more on doing the research we do more systematically instead of jumping from one study to the next.

[Nikhar Gaikwad Yale University]

The guest post quoted below, Taylor's response to it, and Amy's follow-up post all make excellent points. One question that arises from this discussion is whether the TRAX (or a similar type of documentation) should be required of researchers, or whether it should be a recommended practice, or whether it should be allowed to emerge organically (in some form or the other) as a norm. Amy raises a number of concerns about journals requiring such documentation. Taylor's comments and the guest post suggest that at least some (if not many) of these concerns apply to quantitative research as well. The guest post points out that detailed appendices in quantitative research reflect signaling incentives, and that similar incentives may lead to the emergence of detailed appendices in qualitative research over time. Additional perspectives on this topic would be most welcome.

In addition to conducting qualitative research, I also conduct quantitative research. As Ingo suggests, I think some consideration of the rise of the lengthy quantitative appendix is useful when thinking about what our field would look like if we also encouraged the rise of the lengthy qualitative appendix.

Just 5-10 years ago, when I submitted quantitative papers to journals, I did not submit lengthy appendices -- either I did not submit an appendix at all, or I submitted an appendix of only 1-2 pages, which typically described some key facet of variable coding or the sample. Today, when I submit quantitative papers to journals, I always submit an appendix, and this appendix will be 20-40 pages long in the end and often longer than the paper itself. In addition to extensive information about data collection, sampling, measurement, and background on my case(s), my appendix will also contain at least 10-20 additional tables and figures. Any robustness check discussed in the text appears in full in the appendix. This type of appendix is in no way formally required by journals, but my impression is that reviewers increasingly expect to see it from authors. I want to send a signal that I am a careful scholar, so I make a lengthy appendix.

Making this kind of appendix is not trivial. Maybe/probably it is a lot easier than making a TRAX (I've never made one), but believe me, it is NOT easy. It actually takes a lot of work -- which is why it sends a signal about the quality of my paper. And like Sam who commented above, it has been a "very useful exercise in academic self-discipline" for me. I have to essentially treat every robustness check as carefully as I would my main analysis. And since my quantitative research almost always involves original data collection, I have to be much more careful to document every step of the data collection process so that I can report it months/years later in the appendix. I feel more confident in my research as a result. That being said, it takes a lot of time -- and it's not as if people are actually reading my appendix... But I want my research to get published! So I do it.

I think that this signaling dynamic suggests that -- regardless of whether DA-RT or JETS creates formal incentives for qualitative researchers to create a lengthy appendix -- we might start seeing them because of the signaling dynamic I outline above. To again speak from my personal experience: I considered making a TRAX for my most recent qualitative paper and decided against it. I figured that it would be a useful exercise and probably improve the research on the margins. But at the same time, I figured that it would take a lot of time and that no one expects to see one yet. (I've never reviewed a qualitative article that made use of active citation or TRAX.) So I thought the trade-off in terms of my time wasn't worth it.

But I wonder if it will stay that way, regardless of the outcomes of these deliberations. My experience with quantitative research leads me to be cynical/pessimistic. Someone is going to submit an article with an accompanying TRAX, and then someone is going to review it and think, "huh, this is impressive." (You can see how this dynamic might start to emerge from some of the earlier comments on this thread -- even if we don't demand a TRAX, at least some people are going to think it looks good.) And then more people will start sending in TRAX to send a signal about the quality of the research, and it might become the norm or expectation. So I'm not really sure what "we" as a profession can do to stop people from upping the ante in terms of the signal that they're trying to send in terms of their research quality. If we're concerned about inequalities or the time lost to us as qualitative researchers through having to create TRAX (or whatever else), then perhaps part of the solution should be to create systems to make them less costly, especially to under-resourced researchers. Or perhaps journals should create upper limits for appendices shared with reviewers.

[Nikhar Gaikwad Yale University]

A number of scholars weighed in on the costs and benefits of specific transparency practices pertaining to text-based sources in Stage I of the QTD process. I have included below links to some threads that are particularly relevant to the topics discussed in this thread:

Active citation versus the meaty footnote

Against "requirements"

What might qualitative data access look like?

Are qualitative researchers held to a higher standard?

Journal standards in practice

A brighter side to data dissemination

[Veronica Herrera University of Connecticut]

From Zehra Arat, UCONN:

I have considerable reservations about the QTD effort or what the qualitative researchers are trying to do lately. There seems to be too much push for certain kind of empiricism to make qualitative research as “respectable” and as “rigorous” as the quantitative one. Well, some of us turn to qualitative & interpretive methods because we find the quantitative method as inappropriate and inadequate to address the question in hand. By the same token, I do not think we can apply the same measure of transparency and rigor to very distinct methods.

[Guest]

This has been a very thoughtful thread, and I'm sorry to be posting so late.

The following comment (originally posted by a guest on November 15) really resonated with me: "Leaving aside limitations of photocopies or scans, to require a researcher to post images of all of her documents places a huge burden on the researcher -- particularly if they are a junior scholar or graduate student...Instead, archival research would benefit from systematic and consistent footnoting practices that identify all of the pieces of information another researcher would need to find the documents him or herself."

The approach suggested by this commentator (that is, encouraging sufficiently informative footnoting) gets the general benefit-cost calculation right to my mind, especially in the case of archivally intensive research. I am inclined to think of it as the most sensible baseline. If a key claim hinges crucially on the text of a particular document, as is sometimes the case, then perhaps it makes sense to make the document available online in some manner. But this should be done selectively, or else the costs to individual researchers rapidly begin to outstrip the collective benefits to the scholarly community.

Or am I missing some obvious argument that a slight increase of the burden on individual researchers (beyond sufficiently informative footnoting) actually results in a much greater collective benefit in terms of transparency?