III.5. Content analysis (non-automated)

Forum rules

We encourage contributors to the Discussion Board to publicly identify by registering and logging in prior to posting. However, if you prefer, you may post anonymously (i.e. without having your post be attributed to you) by posting without logging in. Anonymous posts will display only after a delay to allow for administrator review. Contributors agree to the QTD Terms of Use.

Instructions
To participate, you may either post a contribution to an existing discussion by selecting the thread for that topic (and then click on "Post Reply") or start a new thread by clicking on "New Topic" below.

For instructions on how to follow a discussion thread by email, click here.

Jonas Tallberg
Stockholm University
Posts: 2
Joined: Thu Sep 01, 2016 7:48 am

Transparency requirements journals

PostMon Sep 19, 2016 9:00 am

What documentation in terms of source materials and coding decisions is it reasonable for journals to demand? What is it unreasonable to demand?

Post Reply

Guest

Re: Transparency requirements journals

PostThu Oct 20, 2016 4:54 am

[quote="jonastall"]What documentation in terms of source materials and coding decisions is it reasonable for journals to demand? What is it unreasonable to demand?[/quote]

Reasonable: codebook, list of sources used, access to source material used for coding (except see below).
Unreasonable: protocols of individual coding decisions, publicly available source material, protected material (anonymity!)

Post Reply

Diana Panke
Albert-Ludwigs University Freiburg
Posts: 2
Joined: Fri Oct 28, 2016 2:18 am

Re: Transparency requirements journals

PostFri Oct 28, 2016 5:42 am

Q: What documentation in terms of source materials and coding decisions is it reasonable for journals to demand? What is it unreasonable to demand?

In my opinion, it is reasonable for journals to demand public accesss of the data used in a paper (for replication purposes only, not the entire database) and the codebook.
If not already included in the codebook, journals should request authors to provide a list of data sources. However, authors should not be asked to provide the material of not publicly accessible sources (e.g. interview material, classified documents), as this would violate ethical codes with respect to the anonymity of the involved actors.
I think it is reasonable for journals to ask authors to explain their coding decisions. Coding and measurement of variables has important implications for the intersubjective interpretation of findings and the potential for generalizations.

Post Reply

Henning Schmidtke
Bavarian School of Public Policy (Technical University of Munich)
Posts: 1
Joined: Tue Nov 01, 2016 6:09 am

Re: Transparency requirements journals

PostTue Nov 01, 2016 6:45 am

Q: What documentation in terms of source materials and coding decisions is it reasonable for journals to demand? What is it unreasonable to demand?

My area of expertise is mainly in newspaper content analysis. I agree with the above authors that it is absolutely reasonable for journals to demand a detailed codebook the makes the entire coding process, that is procedures of text retrieval and text selection, text sources, and coding rules, transparent. If replication in any meaningful way is possible there should also be access to the data set used in a paper. However, this might become very difficult in the field of newspaper content analysis as soon as the data set contains original newspaper material. The reason for this problem is that the terms and conditions of the large digital newspaper archives (e.g. FACTIVA or LEXISNEXIS) include clauses that do not allow you to share the content you obtained. So while quoting a newspaper article here and there in paper might still be ok, sharing, for instance, a larger volume of coded segments of newspaper articles might already be a violation of copyright regulations. So from my point of view there is currently legal grey area and it would be great, if a solution on how to weigh questions of transparent research and copyright issues could be found with large digital newspaper databanks.

In addition, I think journals may also demand for coding reliability information where it is appropriate. I think it’s reasonable to demand for reliability test, when larger volumes of text are coded over a longer time period or when more than one human coder is involved. Information on reliability should than include a discussion of coder training (how and how long have coders learned to use the coding rules) and on formal and informal reliability tests. For formal reliability tests (percent agreement or better Krippendorff’s Alpha) the sample size for which reliability was tested and the respective results should be presented. Where thresholds are not met authors should be required to discuss how they dealt with these problems. Here, I think more informal reliability checks, such as group discussion or double coding by different human codes, could be appropriate solutions.

Henning Schmidtke

Post Reply