Substantive Dimensions of the Deliberations

Forum rules

We encourage contributors to the Discussion Board to publicly identify by registering and logging in prior to posting. However, if you prefer, you may post anonymously (i.e. without having your post be attributed to you) by posting without logging in. Anonymous posts will display only after a delay to allow for administrator review. Contributors agree to the QTD Terms of Use.

Instructions
To participate, you may either post a contribution to an existing discussion by selecting the thread for that topic (and then click on "Post Reply") or start a new thread by clicking on "New Topic" below.

The transition to Stage 2 of the deliberations is currently underway but will take some time to complete. In the meantime, we very much welcome additional contributions to the existing threads in this forum.

For instructions on how to follow a discussion thread by email, click here.

Guest

AJPS badges

PostTue May 10, 2016 9:12 am

AJPS is introducing gamification techniques to promote transparency, articles that comply with their dart policy receive 2 badges: open data and open material (this is not a late April fool it was posted today)

https://ajps.org/2016/05/10/ajps-to-awa ... ce-badges/

the badges are presented by the writer of the article just as a visual cue to easily identify what paper has passed their replication procedures and uploaded the data and code.

Here is the piece of the article that talks about exceptions:

"Of course, some articles published in the American Journal of Political Science will not receive the Badges. For example, many formal theory manuscripts and virtually all of the normative theory manuscripts that are submitted to the Journal do not contain any empirical analyses. Such work is exempt from the AJPS Replication Policy, so the Open Practice Badges are not relevant to these manuscripts. And there are certain situations in which a manuscript may be given an exemption from the usual replication requirements due to the use of restricted-access data. In such cases, authors still are asked to explain how interested researchers could gain access to the data and to provide all relevant software code and documentation for replicating their analyses. Manuscripts in this situation would not receive the Open Data Badge, but they would be awarded the Open Materials Badge. Even with allowances for exceptions, we anticipate that the vast majority of the articles published in the American Journal of Political Science will receive both Badges."

I wonder if they thought that some of the really interesting types of qualitative research that have been discussed in QTD might not get these badges... I guess they might be added to exceptions.

But more generally this type of gamification device is a paternalistic carrot/stick weak incentive structure.

I am open to see how it will affect the journal, and ready to accept that these incentives work and will improve the rate of replication/transparency in quantitative research, but part of me also resists to these types of approaches.

My hunch is that it will be useless for quantitative research, the only outcome will be stigmatizing the exceptions. But I am ready to be proven wrong.

Post Reply

Marcus Kreuzer
Villanova University
Posts: 26
Joined: Sat Apr 16, 2016 9:48 am

Re: AJPS badges

PostWed May 11, 2016 10:59 am

Thanks for pointing out the AJPS badges.
While I share your concerns, the idea of badges is intriguing if the could be earned for more than just data access and replication. Let's assume that you could earn badges for the type of research transparency that is more relevant to qualitative research, they could convey important information and become tools for gradually establishing new transparency norms without enforcing them from upon high. Take a simple example, let's assume that you could earn a citation badge for having more than 70% of your citations including and actual page number. Or you could earn a badge for active citation. Such badges would provide small carrots to improve research transparency. Bottom line, badges are not the problem but what research activities can earn them is.

Post Reply

Paolo Spada
Southampton
Posts: 11
Joined: Wed Apr 06, 2016 8:24 pm

Re: AJPS badges

PostThu May 12, 2016 4:38 am

Hi Markus, I find your idea very interesting: a variety of badges and incentives to promote commonly agreed community enhancing behavior.

But the discussion about top down incentive structure and choice architectures is always tricky. There is a growing debate around these gamification approaches and many view these incentive structures as the dark side of gamification and contrast it with playfulness (see for example all the interesting work of Eric Gordon). I am still on the fence.

Setting aside philosophy and freedom, from a purely instrumental standpoint there are plenty of examples in which incentives backfired.

For example project repurpose of AFL-CIO was born because during the first Obama campaign they a/b tested a badge scheme and discovered that it reduced the motivation of people. Instead repurpose substituted the badges with micro-empowerment. Participants earned points by canvassing that could be spent in targeting the effort of the campaign itself instead of getting a meaningless badge.
[link to repurpose: http://repurpose.workersvoice.org/]

Successful implementation of these schemes more and more use the micro-empowerment approach (for example stack overflow increases the administrative privileges of participants that perform community enhancing activities).

Not sure if there is a way to apply the micro-empowerment approach to journals. Probably in the review track there is, creating categories of super-reviewers that get additional editorial privileges the more they review. In the publication track I am not sure. Maybe the possibility of getting additional word count could be a proper incentive. But all these things need very careful field testing because they have the habit of creating unintended consequences as the infamous Haifa experiment on fines for parents that arrived late to pick up kids at school showed.

And why instead not introducing more participatory decision making from readers/reviewers/authors in journals?

Why for example discussions like this one cannot be promoted among the readers/authors/reviewers of CPS to define their rules?

Maybe we need more deliberation instead off nudges/incentives...

Post Reply

Paolo Spada
Southampton
Posts: 11
Joined: Wed Apr 06, 2016 8:24 pm

Re: AJPS badges

PostThu May 12, 2016 3:09 pm

Five 38 just posted an interesting article about the benefits of badges in academic publishing

http://fivethirtyeight.com/features/eve ... 538twitter

Post Reply

Mala Htun
Univ of New Mexico
Posts: 16
Joined: Wed Apr 06, 2016 9:20 am

Re: AJPS badges

PostFri May 13, 2016 10:33 am

Wow. im digesting all of this. i am immediately concerned about the badges privileging and creating additional incentives for certain types of research that are amenable to immediate sharing of digitized data, while denigrating and dis-incentivizing others. It is hypothetically possible that badges could be awarded, as Marcus says, to articles that include page numbers in footnotes, or that avoid gender bias in citation practices, or that adopt additional methods and approaches to examine the research problem from diverse angles, or whatever. but the very existence of the badge creates a hierarchy between articles that get the badge, and thus are better and more valuable, and those that don't. this strikes against pluralism in social and political inquiry.

Post Reply

Guest

Re: AJPS badges

PostTue May 17, 2016 3:52 pm

I don't really like the idea of badges (I share Mala's concerns) but on the other hand, I do think it is important for qualitative researchers to not reject efforts to improve research transparency out of hand.

Marcus' suggestion of something as simple as requiring more citations with page numbers--- and applying this across qualitative and quantitative work--- actually makes a lot of sense to me. I often read papers where I feel like authors make hand-waves to citations that they have read very superficially at best (else they would have shown a little more nuance in their characterization of a given argument). Taken in moderation, this strategy seems a fairly low-cost way to improve the quality of scholarship. But of course, as with everything else in life, I wouldn't want to see this strategy taken too far. Put more concretely, I'm not sure it's really worth all the extra time/energy to figure out where, precisely, Barrington Moore wrote "no bourgeosie, no democracy."

Post Reply