How will we rate papers in the system?

What should a score be: words or numbers?
Should they relate to the current journal paragidm?
Technicalities v. importance – facet out the score?
– too complicated!
Do we let this emerge?
What kind of ratings would people like – later in the survey?
Discussion board topic on ratings

Jarrett’s Scale
Seriously flawed
Major science concerns
Minor science concerns
Science is good, but poorly written
Science is good, text needs minor cleanup
Ready for publication
Outstanding paper in the field

Cameron’s scale
This paper:
is seriously flawed – 1
has some technical issues – 2
is useful – 3
is a significant contribution – 4
is an outstanding contribution to the field -5

We all agree on Cameron’s scale. The slightly more vague scale actually has some very large advantages.

How does this contribute to Karma?
Reviews do not reduce Karma
But a score of 1 does not add to Karma
Nonlinear function of scores

We need to come up with a scoring formula for papers!

Elements of the Karma System

What are the elements of a reputation/karma system? What sorts of activities should gain/lose karma, and what abilities should they ‘buy’ in a preprint ecosystem? Here are our notes:

Activities Generating Karma

– adding tags
– suggesting tags
– moderating comments
– moderating tags
– commenting
– voting up a comment
– voting down a comment
– having a comment voted up
– having a comment voted down
– posted a paper
– reviewing a paper
– scores from review: review formula – ratings normalized by review score and maybe reviewer
– voting up a review
– voting down a review
– having a review voted up
– having a review voted down
– author upvotes your review
– author upvotes your comment
– moderating users
– using a social media button

Privileges that Karma Unlocks
– Post a paper (0 Karma)
– Post a comment (0 Karma)
– Post a review (0 Karma)

– Vote up a comment
– Vote up a review

– Vote down a comment
– Vote down a review

– Flag a paper
– Flag a tag
– Flag a comment
– Flag a review
– Flag a user

– Add a tag to someone else’s paper

– Create a tag (starts 3-4 monts in)

– moderation priviledges

Starting Ideas for Post-Use Survey of OpenPub

Encourage conversation about the site on the site

Add a feedback email link

What do we want to achieve from a survey?

The purpose/motivation is

1) Increase speed of the scholarly communication process

2) Provide a place for preprints and other works

3) Demonstrate the value of conversation as it relates to scholarly works

4) Demonstrate the value of assigning value to review

5) Demonstrate the value of preprints

General Questions

Did you find all of this useful as an author?

Did you find all of this useful as an reading consumer?

Did you find all of this useful as an actively participating consumer?

Do you now feel more engaged and interested with preprints, conversation, and the review process?

Which of these features are useful?  Which are not? Tear it down to each piece?

What other features would you find useful?

Has this changed you as a participant in the scholarly communications process?

How active are you? As a reader? As a participant? As an author?


Issues to consider

When do we survey people? Email their survey after their account is 6 months old.

Prod people to participate if they haven’t held up their end of the bargain? Don’t show karma until they have done the thing

Surveying Opinions on Scholarly Publishing in EEB

If you want to envision change to the current system of publication, you need to know what people think about where we are, and where we want to go with the future of scholarly publication. So, as part of our working group, today we are launching a survey of attitudes about scholarly publishing and communication by Ecologists, Evolutionary Biologists, and Earth & Environmental Scientists.

Our goal is to establish a baseline of your opinions on the current state of scholarly communication for our fields so as to highlight potential gaps and improvements.

Please, head on over and take the survey. We want your input. We’ll be using it both to write up a paper, and inform a few future projects from the group.

Agenda for Meeting 2

1) Overview of goals (review)
2) Review of progress (everyone be ready to give a 5 minute status update)
3) Mark Talk about ESA meeting
– How can we take advantage of Collins being president
4) Chris talk about the IEE Special Issue
5) Discuss the survey questions and nail it down. Then Chris hits send.


The Eco-Preprint-Commentary Project
– Ed presents what has been done (morning)
– Open discussion on the project

Afternoon, small groups
-designing a post-use survey for the preprint server
– devs continue work based on morning discussion

– plenary bringing survey and devs together, make sure goals are aligned

Afternoon small groups
– devs continue work based on morning discussion
– Strategize roll-out of project and assign roles

– final presentation by developers
– Strategize roll-out and Science Online presentation
– Discuss future administration of project
– Discuss funding of future OpenPub efforts