Page 1 of 1

Transparency requirements journals

Posted: Mon Sep 19, 2016 9:00 am
by jonastall
What documentation in terms of source materials and coding decisions is it reasonable for journals to demand? What is it unreasonable to demand?

Re: Transparency requirements journals

Posted: Thu Oct 20, 2016 4:54 am
by Guest
[quote="jonastall"]What documentation in terms of source materials and coding decisions is it reasonable for journals to demand? What is it unreasonable to demand?[/quote]

Reasonable: codebook, list of sources used, access to source material used for coding (except see below).
Unreasonable: protocols of individual coding decisions, publicly available source material, protected material (anonymity!)

Re: Transparency requirements journals

Posted: Fri Oct 28, 2016 5:42 am
by dianapanke
Q: What documentation in terms of source materials and coding decisions is it reasonable for journals to demand? What is it unreasonable to demand?

In my opinion, it is reasonable for journals to demand public accesss of the data used in a paper (for replication purposes only, not the entire database) and the codebook.
If not already included in the codebook, journals should request authors to provide a list of data sources. However, authors should not be asked to provide the material of not publicly accessible sources (e.g. interview material, classified documents), as this would violate ethical codes with respect to the anonymity of the involved actors.
I think it is reasonable for journals to ask authors to explain their coding decisions. Coding and measurement of variables has important implications for the intersubjective interpretation of findings and the potential for generalizations.

Re: Transparency requirements journals

Posted: Tue Nov 01, 2016 6:45 am
by HenningSchmidtke
Q: What documentation in terms of source materials and coding decisions is it reasonable for journals to demand? What is it unreasonable to demand?

My area of expertise is mainly in newspaper content analysis. I agree with the above authors that it is absolutely reasonable for journals to demand a detailed codebook the makes the entire coding process, that is procedures of text retrieval and text selection, text sources, and coding rules, transparent. If replication in any meaningful way is possible there should also be access to the data set used in a paper. However, this might become very difficult in the field of newspaper content analysis as soon as the data set contains original newspaper material. The reason for this problem is that the terms and conditions of the large digital newspaper archives (e.g. FACTIVA or LEXISNEXIS) include clauses that do not allow you to share the content you obtained. So while quoting a newspaper article here and there in paper might still be ok, sharing, for instance, a larger volume of coded segments of newspaper articles might already be a violation of copyright regulations. So from my point of view there is currently legal grey area and it would be great, if a solution on how to weigh questions of transparent research and copyright issues could be found with large digital newspaper databanks.

In addition, I think journals may also demand for coding reliability information where it is appropriate. I think it’s reasonable to demand for reliability test, when larger volumes of text are coded over a longer time period or when more than one human coder is involved. Information on reliability should than include a discussion of coder training (how and how long have coders learned to use the coding rules) and on formal and informal reliability tests. For formal reliability tests (percent agreement or better Krippendorff’s Alpha) the sample size for which reliability was tested and the respective results should be presented. Where thresholds are not met authors should be required to discuss how they dealt with these problems. Here, I think more informal reliability checks, such as group discussion or double coding by different human codes, could be appropriate solutions.

Henning Schmidtke