What should a score be: words or numbers?
Should they relate to the current journal paragidm?
Technicalities v. importance – facet out the score?
– too complicated!
Do we let this emerge?
What kind of ratings would people like – later in the survey?
Discussion board topic on ratings

Jarrett’s Scale
Seriously flawed
Major science concerns
Minor science concerns
Science is good, but poorly written
Science is good, text needs minor cleanup
Ready for publication
Outstanding paper in the field

Cameron’s scale
This paper:
is seriously flawed – 1
has some technical issues – 2
is useful – 3
is a significant contribution – 4
is an outstanding contribution to the field -5

We all agree on Cameron’s scale. The slightly more vague scale actually has some very large advantages.

How does this contribute to Karma?
Reviews do not reduce Karma
But a score of 1 does not add to Karma
Nonlinear function of scores

We need to come up with a scoring formula for papers!

What are the elements of a reputation/karma system? What sorts of activities should gain/lose karma, and what abilities should they ‘buy’ in a preprint ecosystem? Here are our notes:

Activities Generating Karma

– adding tags
– suggesting tags
– moderating comments
– moderating tags
– commenting
– voting up a comment
– voting down a comment
– having a comment voted up
– having a comment voted down
– posted a paper
– reviewing a paper
– scores from review: review formula – ratings normalized by review score and maybe reviewer
– voting up a review
– voting down a review
– having a review voted up
– having a review voted down
– author upvotes your review
– author upvotes your comment
– moderating users
– using a social media button

Privileges that Karma Unlocks
– Post a paper (0 Karma)
– Post a comment (0 Karma)
– Post a review (0 Karma)
—
– Vote up a comment
– Vote up a review
—
– Vote down a comment
– Vote down a review
—
– Flag a paper
– Flag a tag
– Flag a comment
– Flag a review
– Flag a user
—
– Add a tag to someone else’s paper
—
– Create a tag (starts 3-4 monts in)
—
– moderation priviledges

Encourage conversation about the site on the site

Add a feedback email link

What do we want to achieve from a survey?

The purpose/motivation is

1) Increase speed of the scholarly communication process

2) Provide a place for preprints and other works

3) Demonstrate the value of conversation as it relates to scholarly works

4) Demonstrate the value of assigning value to review

5) Demonstrate the value of preprints

General Questions

Did you find all of this useful as an author?

Did you find all of this useful as an reading consumer?

Did you find all of this useful as an actively participating consumer?

Do you now feel more engaged and interested with preprints, conversation, and the review process?

Which of these features are useful?  Which are not? Tear it down to each piece?

What other features would you find useful?

Has this changed you as a participant in the scholarly communications process?

How active are you? As a reader? As a participant? As an author?

Issues to consider

When do we survey people? Email their survey after their account is 6 months old.

Prod people to participate if they haven’t held up their end of the bargain? Don’t show karma until they have done the thing