II.1. Text-based sources

Forum rules

We encourage contributors to the Discussion Board to publicly identify by registering and logging in prior to posting. However, if you prefer, you may post anonymously (i.e. without having your post be attributed to you) by posting without logging in. Anonymous posts will display only after a delay to allow for administrator review. Contributors agree to the QTD Terms of Use.

Instructions
To participate, you may either post a contribution to an existing discussion by selecting the thread for that topic (and then click on "Post Reply") or start a new thread by clicking on "New Topic" below.

For instructions on how to follow a discussion thread by email, click here.

Nikhar Gaikwad
Yale University
Posts: 15
Joined: Tue May 24, 2016 6:53 am

Documenting use of text-based or non-text-based sources

PostWed Sep 07, 2016 3:57 pm

When you cite your sources, what types of information do you include either in-text or in the associated footnotes/endnotes in order to help readers understand how the evidence you are using supports the analytical, descriptive, or causal claims you are making? What types of information, and how much information, do you provide in order to help readers understand the “analytical logic” you are using to link your sources to your claims?

What are your current practices for documenting how your data was produced? Do you discuss or evaluate the quality of your data? Do you describe the advantages and disadvantages of using particular sources over others? Do you explain how you obtained your sources and where they were located?

Post Reply

Kimberly Morgan
George Washington University
Posts: 3
Joined: Thu Apr 07, 2016 3:51 pm

Re:

PostThu Sep 08, 2016 5:48 pm

This is an interesting question: in political science, I would not say that we have necessarily been trained or encouraged to include these types of observations. There also is not a lot of room, given the word limits of many journals and/or the discouraging of footnotes, to elaborate critically on one's sources. I see three potential questions arising out of Nikhar's question: (1) are qualitative scholars in our field currently being trained to think about these kinds of questions in their research? (2) if not, should they be? (3) should we be trying to find ways in the publication process to enable scholars to include these types of reflections?

Post Reply

Guest

Re: Documenting how text-based or non-text-based sources support your arguments

PostFri Sep 09, 2016 9:52 am

Something that ought to be the bare minimum when citing sources is to include page numbers. I am struck by how common it is for scholars in our field to cite entire books or articles without providing any page numbers. Doing so essentially prevents readers from being able to check and make sure that the original sources actually support the claims that they're supposed to support. Although changing this practice would be a fairly modest improvement in the context of all the larger issues being discussed in this forum, I think it would lead to a very tangible improvement in the transparency of our research. Moreover, it is something "easy" to fix in the sense that we can straightforwardly ask -- when advising graduate student work, reviewing articles, etc. -- for authors to provide page numbers.

Post Reply

Marcus Kreuzer
Villanova University
Posts: 26
Joined: Sat Apr 16, 2016 9:48 am

Re: Re:

PostTue Sep 13, 2016 3:31 pm

kjmorgan wrote:This is an interesting question: in political science, I would not say that we have necessarily been trained or encouraged to include these types of observations. There also is not a lot of room, given the word limits of many journals and/or the discouraging of footnotes, to elaborate critically on one's sources. I see three potential questions arising out of Nikhar's question: (1) are qualitative scholars in our field currently being trained to think about these kinds of questions in their research? (2) if not, should they be? (3) should we be trying to find ways in the publication process to enable scholars to include these types of reflections?


Excellent questions.
I think that most of the "training" comes from the journal submission guidelines instructing authors to minimize footnote and keeping them as brief as possible. This is one of the ironies of the DA-RT initiative. Quantitative scholars are encouraged to submit their data files and codes, while quantitative scholars are actively discouraged from using substantial footnotes. Analytical transparency for un-standardized qualitative evidence will never be reduced to a singular code but needs to be address on a case by case basis with the help of footnotes. If only somebody could let journal editors know ...

Post Reply

Marcus Kreuzer
Villanova University
Posts: 26
Joined: Sat Apr 16, 2016 9:48 am

Re: Documenting how text-based or non-text-based sources support your arguments

PostTue Sep 13, 2016 3:46 pm

Nikhar Gaikwad wrote:When you cite your sources, what types of information do you include either in-text or in the associated footnotes/endnotes in order to help readers understand how the evidence you are using supports the analytical, descriptive, or causal claims you are making? What types of information, and how much information, do you provide in order to help readers understand the “analytical logic” you are using to link your sources to your claims?


Here the three uses of footnotes that I see historians employ:

1) assure data access by pointing the reader to the precise location of evidence. Adding page numbers in those instances should not be considered a courtesy but an imperative.
2) assure production transparency by discussing the broader context from which the piece of evidence was taken. Maybe address issues related to different possible translations of the passage. What other sources I consulted in search of the same piece of evidence but did not find anything. (i.e. speculate what the absence of evidence in places where I would have expected them might mean)
3) assure analytical transparency by clarifying how the tangible piece of evidence supports an inference to a broader, and usually not readily observable claim. Essentially, this involves stating the reasons supporting my interpretation of the evidence. I also might cited other works that support my interpretation or other, related pieces of evidence.
Obviously, historians don't use these three possible functions of footnoting all the time. Even editors of history journals are subject to space constraints. But, these three uses would be commonly found in instances where a piece of evidence is particularly important to a claim or where its interpretation is potentially ambiguous.

Post Reply
Last edited by Marcus Kreuzer on Tue Oct 18, 2016 9:54 pm, edited 1 time in total.

Veronica Herrera
University of Connecticut
Posts: 13
Joined: Wed Aug 31, 2016 8:07 am

Re: Documenting how text-based or non-text-based sources support your arguments

PostTue Oct 18, 2016 8:48 am

I agree that citing page numbers is really important and surprisingly not as widespread a practice as it should be, particularly for studies whose inferential leverage is based on text based sources. This is an excellent "low hanging fruit" suggestion to be adopted widely. As reviewers we should insist on this practice, which should not be viewed as minutiae or nitpicking but rather a critical foundation for greater transparency.


Guest wrote:Something that ought to be the bare minimum when citing sources is to include page numbers. I am struck by how common it is for scholars in our field to cite entire books or articles without providing any page numbers. Doing so essentially prevents readers from being able to check and make sure that the original sources actually support the claims that they're supposed to support. Although changing this practice would be a fairly modest improvement in the context of all the larger issues being discussed in this forum, I think it would lead to a very tangible improvement in the transparency of our research. Moreover, it is something "easy" to fix in the sense that we can straightforwardly ask -- when advising graduate student work, reviewing articles, etc. -- for authors to provide page numbers.

Post Reply

Veronica Herrera
University of Connecticut
Posts: 13
Joined: Wed Aug 31, 2016 8:07 am

Re: Documenting how text-based or non-text-based sources support your arguments

PostTue Oct 18, 2016 9:01 am

Marcus, thank you for these comments. Would you say that historians can pick which combination of these three to emphasize at their discretion (that is, when they believe a piece of evidence may point ambiguously to a claim) or is there a standard protocol? Do reviewers push back at manuscripts that fail to address 1, 2, 3 in a systematic way? Have online appendices become more widespread in history to allow for more room for documenting use of text based sources?

To what extent do political scientists using text based sources use these three criteria consistently? Or would you characterize the use in political science as different, and if so, how?

Marcus Kreuzer wrote:
Nikhar Gaikwad wrote:When you cite your sources, what types of information do you include either in-text or in the associated footnotes/endnotes in order to help readers understand how the evidence you are using supports the analytical, descriptive, or causal claims you are making? What types of information, and how much information, do you provide in order to help readers understand the “analytical logic” you are using to link your sources to your claims?


Here the three uses of footnotes that I see historians employ:

1) assure data access by pointing the reader to the precise location of evidence. Adding page numbers in those instances should not be considered a courtesy but an imperative.
2) assure production transparency by discussing the broader context from which the piece of evidence was taken. Maybe address issues related to different possible translations of the passage. What other sources I consulted in search of the same piece of evidence but did not find anything. (i.e. speculate what the absence of evidence in places where I would have expected them might mean)
3) assure analytical transparency by clarifying how the tangible piece of evidence supports an inference to a broader, and usually not readily observable claim. Essentially, this involves stating the reasons supporting my interpretation of the evidence. I also might cited other works that support my interpretation or other, related pieces of evidence.
Obviously, historians don't use these three possible functions of footnoting all the time. Even editors of history journals are subject to space constraints. But, these three uses would be commonly found in instances where a piece of evidence is particularly important to a claim or where its interpretation is potentially ambiguous.

Post Reply

Marcus Kreuzer
Villanova University
Posts: 26
Joined: Sat Apr 16, 2016 9:48 am

Re: Documenting how text-based or non-text-based sources support your arguments

PostTue Oct 18, 2016 9:56 pm

Yes, my understanding is that historians use discretion in how they employ footnotes rather than following a protocol. Maybe the one exception is there insistence that footnotes have page numbers. By the way, I extrapolated those three uses from Anthony Grafton's delightful "The Footnote: Curious History." This books also makes clear that, just like with DA-RT, that the footnote was not warmly embraced at first and that its transparency objectives evolved with time. This potential analogy between the footnote and DA-RT implies that we should be realistic in what we can expected and patient to give it time to evolve. If you ask me, if we come out of this making the pageless footnote referencing something specific unacceptable, the entire DA-RT exercise will have been worth it. Doesn't mean though that we should not shoot a bit higher.

VeronicaHerrera wrote:Marcus, thank you for these comments. Would you say that historians can pick which combination of these three to emphasize at their discretion (that is, when they believe a piece of evidence may point ambiguously to a claim) or is there a standard protocol? Do reviewers push back at manuscripts that fail to address 1, 2, 3 in a systematic way? Have online appendices become more widespread in history to allow for more room for documenting use of text based sources?

To what extent do political scientists using text based sources use these three criteria consistently? Or would you characterize the use in political science as different, and if so, how?

Marcus Kreuzer wrote:
Nikhar Gaikwad wrote:When you cite your sources, what types of information do you include either in-text or in the associated footnotes/endnotes in order to help readers understand how the evidence you are using supports the analytical, descriptive, or causal claims you are making? What types of information, and how much information, do you provide in order to help readers understand the “analytical logic” you are using to link your sources to your claims?


Here the three uses of footnotes that I see historians employ:

1) assure data access by pointing the reader to the precise location of evidence. Adding page numbers in those instances should not be considered a courtesy but an imperative.
2) assure production transparency by discussing the broader context from which the piece of evidence was taken. Maybe address issues related to different possible translations of the passage. What other sources I consulted in search of the same piece of evidence but did not find anything. (i.e. speculate what the absence of evidence in places where I would have expected them might mean)
3) assure analytical transparency by clarifying how the tangible piece of evidence supports an inference to a broader, and usually not readily observable claim. Essentially, this involves stating the reasons supporting my interpretation of the evidence. I also might cited other works that support my interpretation or other, related pieces of evidence.
Obviously, historians don't use these three possible functions of footnoting all the time. Even editors of history journals are subject to space constraints. But, these three uses would be commonly found in instances where a piece of evidence is particularly important to a claim or where its interpretation is potentially ambiguous.

Post Reply

Nikhar Gaikwad
Yale University
Posts: 15
Joined: Tue May 24, 2016 6:53 am

Re: Documenting use of text-based or non-text-based sources

PostThu Nov 03, 2016 4:25 pm

Thank you, Marcus, for sharing this helpful information about how historians use footnotes in their research. To summarize, it seems as if the one consistent practice is the inclusion of page numbers with references.

I wanted to draw attention to one question that came up repeatedly in the Stage One QTD deliberations: How should scholars navigate the tradeoff between formulating "meaty footnotes" that achieve the goals of data access, production transparency, and analytical transparency and the very punishing word limits of the top journals in our discipline? The AJPS, for example, has a word limit of 10,000 words, while the JOP restricts submissions to no more than 35 pages (including all text, footnotes, endnotes, references, tables, and figures). Given these word/page limits, is it realistic to require scholars to provide footnotes that meet the three goals discussed above? If yes, should journals first increase word limits for manuscripts relying heavily on qualitative sources? Or should authors include meaty footnotes in online appendices or data repositories? Are there any other solutions to this tradeoff between documenting production and analytical transparency and meeting journal word limits?

Post Reply

Nancy Hirschmann
The University of Pennsylvania
Posts: 6
Joined: Wed Apr 06, 2016 1:59 pm

Re: Documenting use of text-based or non-text-based sources

PostFri Nov 04, 2016 5:19 am

I'm entering this discussion from a different subfield of political theory. Work in our subfield also too often provides in-text references simply noting the source without page number. But I find that when the citation format is footnotes, where more detail is permitted, this is less often the case. So the "scientific notation", ironically, makes it easier to permit a relaxing of the demands of specific citation--because indeed, in scientific journals, author and year are all that is cited unless specific material is quoted. If you summarize an article, what page number would you select from that article? So in fairness, sometimes that's appropriate. The problem is that the existence of this option has permitted us to draw on it too often. But there we go again with the double-bind: in being forced to adapt to a different method of citation than is appropriate to our subfield (in text references with virtually no explanatory footnotes) we end up doing something that creates the foundation for us to be criticized as inferior scholars and not "real" political scientists.

Post Reply

Marcus Kreuzer
Villanova University
Posts: 26
Joined: Sat Apr 16, 2016 9:48 am

Re: Documenting use of text-based or non-text-based sources

PostFri Nov 04, 2016 9:12 am

What is epistemologically desirable with regards to research transparency and what is logistically/economically possible at the publishing end do not always line up and create a real conflict for implementing DA-RT. And it certainly seems to be a tension which the original DA-RT guidelines overlooked. But fortunately, there are ready solutions. Increasing word limits, not counting footnotes towards word limits, moving footnotes on-line, or moving exclusively to online publication to reduce costs of physical printing all seem possible solutions. I am not too fond of banishing footnotes into online appendices. Just like an endnote makes access more cumbersome than a same page footnote, so a footnote buried in some lengthy appendix would further impede access. Apologies, but I am a bit of a footnote aficionado and insist on my God-given American right to instant gratification!

Nikhar Gaikwad wrote: Given these word/page limits [of journals], is it realistic to require scholars to provide footnotes that meet the three goals discussed above? If yes, should journals first increase word limits for manuscripts relying heavily on qualitative sources? Or should authors include meaty footnotes in online appendices or data repositories? Are there any other solutions to this tradeoff between documenting production and analytical transparency and meeting journal word limits?

Post Reply

Richard Valelly
Swarthmore College
Posts: 3
Joined: Thu Apr 07, 2016 2:00 pm

Re: Documenting use of text-based or non-text-based sources

PostFri Nov 04, 2016 1:24 pm

Let me identify myself as another proponent of the informative and detailed footnote. I agree wholeheartedly with Marcus's position that the apparatus for the work that we submit or publish is a major resource for powerfully augmenting the transparency of our work. We should give detailed guidance on how to find what we have found. Whenever possible we should also point our readers to online sources since many sources that we work with have become digitized even if they are also in print. One final possibility -- and I'd be curious what people think of this -- is the construction of websites for our articles and books. We can provide links and/or load content that we have scanned and digitized.

Marcus Kreuzer wrote:What is epistemologically desirable with regards to research transparency and what is logistically/economically possible at the publishing end do not always line up and create a real conflict for implementing DA-RT. And it certainly seems to be a tension which the original DA-RT guidelines overlooked. But fortunately, there are ready solutions. Increasing word limits, not counting footnotes towards word limits, moving footnotes on-line, or moving exclusively to online publication to reduce costs of physical printing all seem possible solutions. I am not too fond of banishing footnotes into online appendices. Just like an endnote makes access more cumbersome than a same page footnote, so a footnote buried in some lengthy appendix would further impede access. Apologies, but I am a bit of a footnote aficionado and insist on my God-given American right to instant gratification!

Nikhar Gaikwad wrote: Given these word/page limits [of journals], is it realistic to require scholars to provide footnotes that meet the three goals discussed above? If yes, should journals first increase word limits for manuscripts relying heavily on qualitative sources? Or should authors include meaty footnotes in online appendices or data repositories? Are there any other solutions to this tradeoff between documenting production and analytical transparency and meeting journal word limits?

Post Reply

Guest

Re: Documenting use of text-based or non-text-based sources

PostTue Nov 08, 2016 5:25 pm

[quote="Nikhar Gaikwad"]Given these word/page limits, is it realistic to require scholars to provide footnotes that meet the three goals discussed above? If yes, should journals first increase word limits for manuscripts relying heavily on qualitative sources? Or should authors include meaty footnotes in online appendices or data repositories? Are there any other solutions to this tradeoff between documenting production and analytical transparency and meeting journal word limits?[/quote]

I do not think that journals' word limits necessarily prevent us from achieving "data access" (goal #1). Page numbers and, where necessary, additional information can be provided without much extra space in footnotes or -- in the case of page numbers -- even in in-text citations. So I do think that page numbers are a minimum goal that we can shoot towards. I like the idea of encouraging online appendices with the "meaty footnotes" and other information that would achieve "production transparency" and "analytical transparency."

Post Reply

Guest

Re: Documenting use of text-based or non-text-based sources

PostTue Nov 15, 2016 4:51 pm

I agree with Marcus' suggestions about increasing transparency by following the examples set by historians. Two additional thoughts:
1) For those of us conducting in depth archival research, we should include all of the information that another researcher would need to identify the source. My archival experience is with the US National archives, and the pertinent pieces of information there are: record group, box number, folder number, type of document (e.g., airgram, telegram, intelligence report), date of document, author/receiver. Depending on the specific archive's organizational system, these information pieces will be different, but consistent and systematic reporting of the location of sources should guard against accusations of cherry picking by providing some context for the document, and will also allow others to follow up on our research.

2) Though encouraging scholars to make available scans or images they have taken of documents, I am wary of the burden this places on researchers using archives, particularly given the amount of time it already takes to identify, collect, and analyze archival materials. First, some archives do not allow you to take pictures, or if they do, do not allow you to make them publicly available. Second, to photograph, sort, and process all of the images (assuming you even took scans of every single document) places an overwhelming burden on the researcher. Though it might be possible for senior scholars to hire research assistants to take thousands of photos, junior scholars and graduate students do not have such resources.

Post Reply

Tasha Fairfield
LSE
Posts: 17
Joined: Mon Sep 05, 2016 4:05 pm

Re: Documenting use of text-based or non-text-based sources

PostSat Nov 19, 2016 6:12 pm

We have an interesting post stream recently started on the comparative/process tracing blog that it quite relevant to the discussion ongoing here and would welcome further comments. I'm taking the liberty of quoting PAHall's post here as well:

"Surely a concern for all non-statistical studies in political science, although perhaps especially so in cross-national comparative studies, is the role that background knowledge (of the history, political context, culture, etc.) of the national/regional case plays in the causal inferences that are drawn from the observations. For example, even if a (hypothetical) French civil servant said to me roughly the same thing as a British civil servant, I might give more credence or weight to one than the other based on this contextual knowledge. I have difficulty seeing how this can be adequately summarized in a transparency index when it applies to what are often hundreds of such observations relevant to the conclusions about a specific case, even if the role of this background knowledge can be discussed for a few key observations. My broader point is that there is invariably an interpretive element to qualitative research that turns heavily on are often complex features of the background context. To apply criteria that might (?) make some sense when the causal inferences are made largely on the basis of counting up various kinds of observations and noting whey they are pertinent to qualitative studies in which what counts most is the quality of those observations -- and a quality that is determined based on deep contextual knowledge -- seems to me not to make sense."

I would add that my own view is that any highly relevant background knowledge that informs our analytical judgements, along with key pieces of evidence on which we base our analysis, should be highlighted in the text of an article. Additional material could be placed in an appendix as needed to deal with word limit issues, but I think we should be wary of providing volumes of additional information in a TRAX purely for the sake of providing lists of additional information. The costs in time and effort may grow quickly with very limited gains, as others have mentioned.

Post Reply

Guest

Re: Documenting use of text-based or non-text-based sources

PostSun Nov 20, 2016 3:46 pm

It seems that this conversation, and a lot of others that I personally have had or witnessed, center on this balance between transparency and parsimony. We want our research to be as parsimonious as possible and with the reality of word and page limits from many of the top academic journals looming over our heads, it can make it difficult to justify our methodological choices. However, we also want to be as transparent as possible to meet a perceived threshold of good social science. I think we need to start looking outside the confinements of the major academic journals in our respective fields for an answer.

There are two ways we can go about this. One, we can develop increasingly formalized practices as we are doing in many cases (such as incorporating saturation, recording methods, response rate, confidence levels etc. as they pertain to interview methods). The drawback of this is that we lose a lot of that nuance (the historical, contextual, socio-political, and normative reasons why we did what we did) because space is just so limiting. This is where my second point comes into play. That is, we should as a social science community start to create the forums necessary where we can broadcast our methods to everyone who is interested. This may include creating links within articles that send the reader to another source where the floor is ours to discuss as much or as little about what we actually did in the archives, in the field, or with interviews. Therefore, the conversation about our data and what we did with it is treated as a source in and of itself so that it takes up no more than one or two sentences in our actual final product but link to a wider conversation that qualitative methodologists have with regards to our different techniques (be they textual analysis, interviews, surveys, or other forms of field work).

Post Reply

abbey steele
university of amsterdam
Posts: 3
Joined: Fri Apr 08, 2016 8:00 am

Re: Documenting use of text-based or non-text-based sources

PostThu Nov 24, 2016 10:19 am

I also agree that the development of citation guidelines for archival sources should be emphasized. For work in archives in developing countries and/or in conflict settings, where I have done research, is much more difficult to cite well. I found archives that were quite disorganized and did my best to cite them in a way that another person could find them later (in consultation with a historian). But there were no page numbers on the documents I found. As an alternative, I did take photographs of the documents and plan to store them at Syracuse University's depository, but I am a junior scholar and even though I took these years ago, I haven't yet spent the time to organize them in a way that would convey their origin well (for future citations). I keep meaning to do it though, because this was a remote archive and I believe the documents could be valuable to others, not only for checking the validity of my inferences.

Guest wrote:
1) For those of us conducting in depth archival research, we should include all of the information that another researcher would need to identify the source. My archival experience is with the US National archives, and the pertinent pieces of information there are: record group, box number, folder number, type of document (e.g., airgram, telegram, intelligence report), date of document, author/receiver. Depending on the specific archive's organizational system, these information pieces will be different, but consistent and systematic reporting of the location of sources should guard against accusations of cherry picking by providing some context for the document, and will also allow others to follow up on our research.

2) Though encouraging scholars to make available scans or images they have taken of documents, I am wary of the burden this places on researchers using archives, particularly given the amount of time it already takes to identify, collect, and analyze archival materials. First, some archives do not allow you to take pictures, or if they do, do not allow you to make them publicly available. Second, to photograph, sort, and process all of the images (assuming you even took scans of every single document) places an overwhelming burden on the researcher. Though it might be possible for senior scholars to hire research assistants to take thousands of photos, junior scholars and graduate students do not have such resources.

Post Reply

Veronica Herrera
University of Connecticut
Posts: 13
Joined: Wed Aug 31, 2016 8:07 am

Re: Documenting use of text-based or non-text-based sources

PostMon Nov 28, 2016 5:13 pm

Thanks for sharing this Abbey. I think many of us have been in this same situation, dealing with unorganized archives (or even loads of boxes from a helpful public servant where no archive exists, sometimes even from their personal notes, etc). That is, developing country research usually necessitates another layer of disorganization and complexity in organizing historic or relevant info, and there is a lot to be said for organizing it somehow for future scholars. If this type of information could be available on our websites or some other way to make it available to a wider audience, it can serve as a public good as well as enhancing transparency. Of course there are always time constraints to consider, but a valuable endeavor nonetheless....




steeleaa wrote:I also agree that the development of citation guidelines for archival sources should be emphasized. For work in archives in developing countries and/or in conflict settings, where I have done research, is much more difficult to cite well. I found archives that were quite disorganized and did my best to cite them in a way that another person could find them later (in consultation with a historian). But there were no page numbers on the documents I found. As an alternative, I did take photographs of the documents and plan to store them at Syracuse University's depository, but I am a junior scholar and even though I took these years ago, I haven't yet spent the time to organize them in a way that would convey their origin well (for future citations). I keep meaning to do it though, because this was a remote archive and I believe the documents could be valuable to others, not only for checking the validity of my inferences.

Guest wrote:
1) For those of us conducting in depth archival research, we should include all of the information that another researcher would need to identify the source. My archival experience is with the US National archives, and the pertinent pieces of information there are: record group, box number, folder number, type of document (e.g., airgram, telegram, intelligence report), date of document, author/receiver. Depending on the specific archive's organizational system, these information pieces will be different, but consistent and systematic reporting of the location of sources should guard against accusations of cherry picking by providing some context for the document, and will also allow others to follow up on our research.

2) Though encouraging scholars to make available scans or images they have taken of documents, I am wary of the burden this places on researchers using archives, particularly given the amount of time it already takes to identify, collect, and analyze archival materials. First, some archives do not allow you to take pictures, or if they do, do not allow you to make them publicly available. Second, to photograph, sort, and process all of the images (assuming you even took scans of every single document) places an overwhelming burden on the researcher. Though it might be possible for senior scholars to hire research assistants to take thousands of photos, junior scholars and graduate students do not have such resources.

Post Reply

Veronica Herrera
University of Connecticut
Posts: 13
Joined: Wed Aug 31, 2016 8:07 am

Re: Documenting use of text-based or non-text-based sources

PostTue Nov 29, 2016 3:36 pm

Thanks for these comments Rick. There are some efforts to create data repositories for qualitative data, for example, https://qdr.syr.edu/. But perhaps something that is hosted on individual websites or online journal formats would make this practice more widespread. What might be some barriers to doing this? Obviously time is a big one, although this is something that research assistants may be able to help with in some cases...I'd be curious to hear from others if they see direct access to some digitalized sources hosted by individual or journal websites as promising, and if so, what may be some costs or barriers that we could think about, and possible solutions?



rvalell1 wrote:Let me identify myself as another proponent of the informative and detailed footnote. I agree wholeheartedly with Marcus's position that the apparatus for the work that we submit or publish is a major resource for powerfully augmenting the transparency of our work. We should give detailed guidance on how to find what we have found. Whenever possible we should also point our readers to online sources since many sources that we work with have become digitized even if they are also in print. One final possibility -- and I'd be curious what people think of this -- is the construction of websites for our articles and books. We can provide links and/or load content that we have scanned and digitized.

Marcus Kreuzer wrote:What is epistemologically desirable with regards to research transparency and what is logistically/economically possible at the publishing end do not always line up and create a real conflict for implementing DA-RT. And it certainly seems to be a tension which the original DA-RT guidelines overlooked. But fortunately, there are ready solutions. Increasing word limits, not counting footnotes towards word limits, moving footnotes on-line, or moving exclusively to online publication to reduce costs of physical printing all seem possible solutions. I am not too fond of banishing footnotes into online appendices. Just like an endnote makes access more cumbersome than a same page footnote, so a footnote buried in some lengthy appendix would further impede access. Apologies, but I am a bit of a footnote aficionado and insist on my God-given American right to instant gratification!

Nikhar Gaikwad wrote: Given these word/page limits [of journals], is it realistic to require scholars to provide footnotes that meet the three goals discussed above? If yes, should journals first increase word limits for manuscripts relying heavily on qualitative sources? Or should authors include meaty footnotes in online appendices or data repositories? Are there any other solutions to this tradeoff between documenting production and analytical transparency and meeting journal word limits?

Post Reply

Veronica Herrera
University of Connecticut
Posts: 13
Joined: Wed Aug 31, 2016 8:07 am

Re: Documenting use of text-based or non-text-based sources

PostTue Nov 29, 2016 3:40 pm

I think your second comment about links for article links up with Rick Valelley's mention of links within articles. I wonder if others agree that this is a good direction to head in, and if so, where these would best be hosted, and what types of guidelines should apply? Or do others disagree with this position? Curious to hear more...


Guest wrote:It seems that this conversation, and a lot of others that I personally have had or witnessed, center on this balance between transparency and parsimony. We want our research to be as parsimonious as possible and with the reality of word and page limits from many of the top academic journals looming over our heads, it can make it difficult to justify our methodological choices. However, we also want to be as transparent as possible to meet a perceived threshold of good social science. I think we need to start looking outside the confinements of the major academic journals in our respective fields for an answer.

There are two ways we can go about this. One, we can develop increasingly formalized practices as we are doing in many cases (such as incorporating saturation, recording methods, response rate, confidence levels etc. as they pertain to interview methods). The drawback of this is that we lose a lot of that nuance (the historical, contextual, socio-political, and normative reasons why we did what we did) because space is just so limiting. This is where my second point comes into play. That is, we should as a social science community start to create the forums necessary where we can broadcast our methods to everyone who is interested. This may include creating links within articles that send the reader to another source where the floor is ours to discuss as much or as little about what we actually did in the archives, in the field, or with interviews. Therefore, the conversation about our data and what we did with it is treated as a source in and of itself so that it takes up no more than one or two sentences in our actual final product but link to a wider conversation that qualitative methodologists have with regards to our different techniques (be they textual analysis, interviews, surveys, or other forms of field work).

Post Reply

Veronica Herrera
University of Connecticut
Posts: 13
Joined: Wed Aug 31, 2016 8:07 am

Re: Documenting use of text-based or non-text-based sources

PostTue Nov 29, 2016 3:42 pm

Thanks for sharing Tasha. These notes are very helpful in moving the conversation forward.
Tasha Fairfield wrote:We have an interesting post stream recently started on the comparative/process tracing blog that it quite relevant to the discussion ongoing here and would welcome further comments. I'm taking the liberty of quoting PAHall's post here as well:

"Surely a concern for all non-statistical studies in political science, although perhaps especially so in cross-national comparative studies, is the role that background knowledge (of the history, political context, culture, etc.) of the national/regional case plays in the causal inferences that are drawn from the observations. For example, even if a (hypothetical) French civil servant said to me roughly the same thing as a British civil servant, I might give more credence or weight to one than the other based on this contextual knowledge. I have difficulty seeing how this can be adequately summarized in a transparency index when it applies to what are often hundreds of such observations relevant to the conclusions about a specific case, even if the role of this background knowledge can be discussed for a few key observations. My broader point is that there is invariably an interpretive element to qualitative research that turns heavily on are often complex features of the background context. To apply criteria that might (?) make some sense when the causal inferences are made largely on the basis of counting up various kinds of observations and noting whey they are pertinent to qualitative studies in which what counts most is the quality of those observations -- and a quality that is determined based on deep contextual knowledge -- seems to me not to make sense."

I would add that my own view is that any highly relevant background knowledge that informs our analytical judgements, along with key pieces of evidence on which we base our analysis, should be highlighted in the text of an article. Additional material could be placed in an appendix as needed to deal with word limit issues, but I think we should be wary of providing volumes of additional information in a TRAX purely for the sake of providing lists of additional information. The costs in time and effort may grow quickly with very limited gains, as others have mentioned.

Post Reply

Shamira Gelbman
Wabash College
Posts: 4
Joined: Tue May 24, 2016 4:40 pm

Re: Documenting use of text-based or non-text-based sources

PostThu Dec 01, 2016 9:52 am

VeronicaHerrera wrote: What might be some barriers to doing this? Obviously time is a big one, although this is something that research assistants may be able to help with in some cases...


Time and access to research assistants are huge, and affect different members of the discipline in different ways. Web space also isn't always free and unlimited.

Finally, creating an expectation that source materials will be digitized opens a whole can of worms with respect to permissions and copyright. Much as it seems like a nice service to make archival materials freely available to everyone, some archives have policies against it (whether generally or for specific collections) and can make it harder for researchers to gain access in the first place if they come to believe that facsimiles of documents will be shared in the peer review and publication process.

Post Reply

Guest

Re: Documenting use of text-based or non-text-based sources

PostFri Dec 09, 2016 10:56 am

Following up on my earlier comments thanks to Abbey's reflection on using archives in developing countries (I'm the same "Guest" from earlier).
I again question whose responsibility it is to provide a digital repository of materials, particularly if the materials come from an unorganized archive. We are NOT archivists -- thus, the way I may choose to digitally organize my files is going to reflect my interpretations and implicit biases about how the documents are situated in relation to one another. That's not necessarily a bad thing, since all archives reflect the quirks and views of their collectors/organizers, but it does make me transition from researcher-scholar to archivist, something that I am not trained for and that I, quite frankly, have no interest in becoming.

Even when we're working in archives from developed countries, where there is an organization scheme (however convoluted), are we expected to reproduce the documents online in full? For example, in my latest trip to the US national archives, I read through approximately 275 boxes' worth of material. There are folders within the boxes and documents within the folders. The totality of documents was essential for me to understand tone, to understand the importance of different characters, and to pick up important but unwritten political context. And yet, the documents I actually quote from represent a fraction of the materials I read through. So would it be -- under the notion of public good or transparency -- my responsibility to come away from the archives with photographs of all of the materials from all 275 boxes, and to put those materials online? Or am I just to make available photographs of the documents I explicitly used, even if such documents would be puzzling to a researcher who had not spent the time reading through everything? If the latter option, then I fail to see how making archival materials digitally available on my website or a public depository is beneficial to anyone. If the former, then I think we need to reflect on data availability in quantitative analyses. That is: the quant datasets uploaded contain ONLY the variables used in that paper/book -- so even if the researcher has 20 other IVs, he/she doesn't make the entirety of the dataset available. This is because (a) it's time consuming to do so, and (b) that way the original researcher still has time to produce more work with the data he/she collected. So there's an argument that parallels DART that would suggest it is not our responsibility to upload everything we read in an archive. To upload just documents we explicitly use would result in random collections of documents available online, from different researchers and hosted in different places. Any attempt to collect and organize photos from archives from different researchers again places us in the role of archivist. But more importantly, I'd suggest that it would be impossible for a second researcher to understand the significance of document #10 without reading documents #1-9, and #11-50.

I think the discussion of the practical barriers to digitally reproducing archival materials is important. But I also think it's the easier conversation to have because it assumes that seeing documents will alleviate others' uneasiness and unfamiliarity with text-based research. We are not archivists. For others to assume that we can and should take on that role fundamentally misunderstands how textual evidence is collected and used to support empirical political science research.

Post Reply

Guest

Re: Documenting use of text-based or non-text-based sources

PostThu Dec 22, 2016 10:43 am

This is Taylor Boas from Boston University, posting as "Guest" since I don't have a login. I'm just reviewing this discussion now. There are a lot of ideas circulating, but I think one of the clearest concerns how to convey details regarding textual sources (or any type of non-library source) given space constraints. In my mind, the answer is clear: online appendices. These have become almost de rigueur in quantitative research given the vast number of alternative specifications, robustness tests, and so on that scholars are expected to present, but which don't fit in the main body. The same could easily be done for details on qualitative sources. Many journals host appendices on their websites, with a link right next to the full-text link for the paper. Since these are non-copyrighted, they can be legally duplicated on a scholar's own personal space (and, in the case of journals that don't host appendices, a scholar's personal web space provides an alternate home). Those publishing in journals that don't host appendices and who lack web space and don't want to pay for it could upload an appendix to a repository such as Dataverse.
Marcus Kreuzer points out that detail in Appendices will be less accessible than footnotes, which is true, but one can minimize the burden with a) a short footnote (or textual mention) in the article stating that more detail is in the Appendix; b) a clearly stated location for the Appendix (e.g., URL of the author's website) in the article's "page 1" footnote alongside acknowledgments and such; c) a detailed table of contents in the Appendix itself. Surely those seeking to visit an archive and examine the same material as the author will be willing to take the extra time to look up additional details in a separate document online.

Post Reply

Guest

Re: Documenting use of text-based or non-text-based sources

PostMon Dec 26, 2016 1:49 pm

For text-based sources, what do you imagine having in the online appendix?
Would it be a description of the sources used and an extended discussion of how the scholar assessed the evidentiary value of different documents or authors?
Would it be a way to shorten footnotes in the article itself and provide full documentation online?
Would it include images of documents or other text sources?

I think the online appendix is an intriguing idea, but am not sure if it solves the fundamental problem that the assumption is that all scholars have the time/resources to take scans or images of all their documents, nor does it answer whether such an endeavor actually makes our work more "transparent."

[quote="Guest"]This is Taylor Boas from Boston University, posting as "Guest" since I don't have a login. I'm just reviewing this discussion now. There are a lot of ideas circulating, but I think one of the clearest concerns how to convey details regarding textual sources (or any type of non-library source) given space constraints. In my mind, the answer is clear: online appendices. These have become almost de rigueur in quantitative research given the vast number of alternative specifications, robustness tests, and so on that scholars are expected to present, but which don't fit in the main body. The same could easily be done for details on qualitative sources. Many journals host appendices on their websites, with a link right next to the full-text link for the paper. Since these are non-copyrighted, they can be legally duplicated on a scholar's own personal space (and, in the case of journals that don't host appendices, a scholar's personal web space provides an alternate home). Those publishing in journals that don't host appendices and who lack web space and don't want to pay for it could upload an appendix to a repository such as Dataverse.
Marcus Kreuzer points out that detail in Appendices will be less accessible than footnotes, which is true, but one can minimize the burden with a) a short footnote (or textual mention) in the article stating that more detail is in the Appendix; b) a clearly stated location for the Appendix (e.g., URL of the author's website) in the article's "page 1" footnote alongside acknowledgments and such; c) a detailed table of contents in the Appendix itself. Surely those seeking to visit an archive and examine the same material as the author will be willing to take the extra time to look up additional details in a separate document online.[/quote]

Post Reply

Nikhar Gaikwad
Yale University
Posts: 15
Joined: Tue May 24, 2016 6:53 am

Re: Documenting use of text-based or non-text-based sources

PostMon Jan 02, 2017 12:35 pm

Thank you for these very helpful suggestions, Taylor.

Guest wrote:This is Taylor Boas from Boston University, posting as "Guest" since I don't have a login. I'm just reviewing this discussion now. There are a lot of ideas circulating, but I think one of the clearest concerns how to convey details regarding textual sources (or any type of non-library source) given space constraints. In my mind, the answer is clear: online appendices. These have become almost de rigueur in quantitative research given the vast number of alternative specifications, robustness tests, and so on that scholars are expected to present, but which don't fit in the main body. The same could easily be done for details on qualitative sources. Many journals host appendices on their websites, with a link right next to the full-text link for the paper. Since these are non-copyrighted, they can be legally duplicated on a scholar's own personal space (and, in the case of journals that don't host appendices, a scholar's personal web space provides an alternate home). Those publishing in journals that don't host appendices and who lack web space and don't want to pay for it could upload an appendix to a repository such as Dataverse.
Marcus Kreuzer points out that detail in Appendices will be less accessible than footnotes, which is true, but one can minimize the burden with a) a short footnote (or textual mention) in the article stating that more detail is in the Appendix; b) a clearly stated location for the Appendix (e.g., URL of the author's website) in the article's "page 1" footnote alongside acknowledgments and such; c) a detailed table of contents in the Appendix itself. Surely those seeking to visit an archive and examine the same material as the author will be willing to take the extra time to look up additional details in a separate document online.

Post Reply

Nikhar Gaikwad
Yale University
Posts: 15
Joined: Tue May 24, 2016 6:53 am

Re: Documenting use of text-based or non-text-based sources

PostMon Jan 02, 2017 1:36 pm

I would like to follow up on this important set of questions. It is worth distinguishing between two forms of transparency-related online documentation.

The first is the online appendix. As Taylor Boas mentioned, online appendices can afford qualitative and multi-methods scholars facing publishing-related word limits with the space to provide detailed context on their sources. In cases where journal word limits preclude scholars from including lengthy footnotes, scholars can provide shorter footnotes within the text of the article, while also directing readers to an online appendix containing additional information related to data access, production transparency, and analytical transparency. One drawback about this practice, as some discussants have pointed out above, is that it impedes access to information; readers are required to navigate a second and potentially cumbersome online document to get access to additional contextual information regarding data sources. Taylor points out that this might not be too big a cost, since only interested readers (for example, those seeking to revisit the source documentation in the relevant archives) would need to review the online appendix. By contrast, general interest readers would be able to access abridged information about the article's sources within the main article itself. An alternate approach, as suggested by Marcus Kreuzer above, would be to require journals to increase word limits so that lengthy footnotes can be included within the text of the manuscripts.

The second form of online documentation pertains to a replication repository. This would be analogous to the replication dataset that most journals now require as a condition for publishing quantitative studies. The initial suggestion here was for qualitative scholars to provide copies (scans, photographs, etc.) of the source materials that they rely upon for making evidentiary claims. We have already heard a number of opinions on this board about the pros and cons of creating replication repositories for qualitative and archival materials. For example, some archives have strict copyright policies that preclude the dissemination of original sources. Additionally, the time and effort required for digitizing and circulating certain forms of qualitative sources might place an undue burden on scholars, especially junior scholars and graduate students.

My sense is that it is worth evaluating the merits and demerits of both forms of documentation separately. As the guest post below aptly notes, the motivating question for us to ask is whether and how these practices help make our work more transparent. Please refer to the discussion on the benefits and costs of transparency for additional opinions related on this topic (see, e.g., https://www.qualtd.net/viewtopic.php?f=17&t=140#p887 and https://www.qualtd.net/viewtopic.php?f=17&t=140#p795)

Guest wrote:For text-based sources, what do you imagine having in the online appendix?
Would it be a description of the sources used and an extended discussion of how the scholar assessed the evidentiary value of different documents or authors?
Would it be a way to shorten footnotes in the article itself and provide full documentation online?
Would it include images of documents or other text sources?

I think the online appendix is an intriguing idea, but am not sure if it solves the fundamental problem that the assumption is that all scholars have the time/resources to take scans or images of all their documents, nor does it answer whether such an endeavor actually makes our work more "transparent."

Guest wrote:This is Taylor Boas from Boston University, posting as "Guest" since I don't have a login. I'm just reviewing this discussion now. There are a lot of ideas circulating, but I think one of the clearest concerns how to convey details regarding textual sources (or any type of non-library source) given space constraints. In my mind, the answer is clear: online appendices. These have become almost de rigueur in quantitative research given the vast number of alternative specifications, robustness tests, and so on that scholars are expected to present, but which don't fit in the main body. The same could easily be done for details on qualitative sources. Many journals host appendices on their websites, with a link right next to the full-text link for the paper. Since these are non-copyrighted, they can be legally duplicated on a scholar's own personal space (and, in the case of journals that don't host appendices, a scholar's personal web space provides an alternate home). Those publishing in journals that don't host appendices and who lack web space and don't want to pay for it could upload an appendix to a repository such as Dataverse.
Marcus Kreuzer points out that detail in Appendices will be less accessible than footnotes, which is true, but one can minimize the burden with a) a short footnote (or textual mention) in the article stating that more detail is in the Appendix; b) a clearly stated location for the Appendix (e.g., URL of the author's website) in the article's "page 1" footnote alongside acknowledgments and such; c) a detailed table of contents in the Appendix itself. Surely those seeking to visit an archive and examine the same material as the author will be willing to take the extra time to look up additional details in a separate document online.

Post Reply

Nikhar Gaikwad
Yale University
Posts: 15
Joined: Tue May 24, 2016 6:53 am

Re: Documenting use of text-based or non-text-based sources

PostMon Jan 02, 2017 4:44 pm

Several scholars weighed in on practices related to the documentation of text-based sources in Stage I of the QTD process. I have included below links to some posts that are relevant to the topics discussed in this thread:

Active citation versus the meaty footnote: Response 1, Response 2, Response 3, Response 4, Response 5, Response 6, Response 7, Response 8

DA-RT: effect on graduate training

Graduate student concerns

Data access--discourage original data production?

Who are the gatekeepers? Editors or reviewers?

Data access and “right to first use” of newly collected quantitative data

Post Reply