the reasons for ranking

[teefal]

Hi all,

Let's talk more directly about the reasons for ranking . . . early
today I wrote:

[yoshiki]

At Tue, 29 Sep 2009 20:24:39 -0400,
Timothy Falconer wrote:

[teefal]

On Sep 29, 2009, at 9:08 PM, Yoshiki Ohshima wrote:

[yoshiki]

At Tue, 29 Sep 2009 22:45:08 -0400,
Timothy Falconer wrote:

[yoshiki]

At Tue, 29 Sep 2009 20:08:46 -0700,
Yoshiki Ohshima wrote:

[teefal]

On Sep 29, 2009, at 11:08 PM, Yoshiki Ohshima wrote:

[voiklis]

Hello all,

I apologize for joining this conversation so late; I am a longtime, though recently quiet, member of the community.

I understand that the ranking system is an attempt to enable people to judge the quality of a contribution (or contributor) based on some directly observable measure of reputation. On first sight, one could argue that such a system overcomes the problem that in the face-to-face world reputation is not directly observable. In order to get an honest assessment of a person's reputation, one has to invest a lot of time building trust with the people familiar with that person. Even then, a trustworthy assessment requires direct observation of the person's actions.

A ranking system would appear to reduce that effort by half; knowing the person's reputation among his/her peers, one only needs to assess the work.

The problem with that reasoning is that electronic ranking systems are highly susceptible to manipulation. Building reputation becomes the goal of the activity for many people and they use all sorts of seemingly harmless social and technological means to inflate their numbers. Our lab has been studying this phenomenon through both observational studies of online communities and laboratory experiments. The two papers below report on the phenomenon as it presents itself in Digg, the news aggregation site.

The first paper demonstrates that a tit-for-tat game of reciprocity inflates the reputation of contributors and their contributions without reflecting anything substantive about their contributions. The second paper really brings out the negative consequences of this phenomenon. The paper reports on an experiment where people judged how interesting they found the contribution. The ranking values of the articles were set by the investigator; sometime the rank of the article was set high, at other times low. Experimental subjects rated higher-ranked contributions as more interesting than lower ranked contributions. The same article was rated as highly interesting when its rank was set high and uninteresting when ranked low. Duncan Watts (of small-world networks fame) observed the same phenomenon with music rating.

What this means in the present discussion is that people will likely ignore low ranking contributions. Worse still, when they do actually look at those contributions they are likely to see what the ranking value led them to expect rather than the qualities of the contribution itself.

Unless we can find scientific research that demonstrates any benefits to ranking, I think we should be wary of using such systems.

All best,

John

Sadlon, E., Sakamoto, Y., Dever, H. J., & Nickerson, J. V. (2008). The karma of Digg: Reciprocity in online social networks. In R. Gopal and R. Ramesh (Eds.), Proceedings of the 18th Annual Workshop on Information Technologies and Systems. http://cog.mgnt.stevens-tech.edu/~yasu/papers/reciprocity.pdf

Sakamoto, Y., Ma, J., & Nickerson, J. V. (2009). 2377 people like this article: The influence of others' decisions on yours. In N. Taatgen, H. van Rijn, L. Schomaker, and J. Nerbonne (Eds.), Proceedings of the 31st Annual Conference of the Cognitive Science Society. http://cog.mgnt.stevens-tech.edu/~yasu/papers/cogscidigg1.pdf

Salganik, M. J., Dodds, P. S., and Watts, D. J. (2006). Experimental study of inequality and unpredictability in an artificial cultural market. Science, 311(5762):854-856. http://dx.doi.org/10.1126/science.1121066[/url]

[teefal]

On Sep 30, 2009, at 10:58 AM, voiklis wrote:

[Rita Freudenberg]

Am 30.09.2009 um 17:20 schrieb Kim Rose:

[yoshiki]

At Wed, 30 Sep 2009 08:44:47 -0400,
Timothy Falconer wrote:

[voiklis]

Hi Kim,

I _am_ well and still very much interested, even if my work has led in
a very different direction. I fully understand the community's need
for some kind of assessment system. I seem to remember that something
of the sort was a topic of conversation at the very first Squeakfest.
I am glad to see that Tim has modeled the ranking/trust system like
some kind of peer review. Nevertheless, I am still wary of whether it
will work or work well; peer review is problematic even in the
sciences. Some kind of "expert" assessment or assessment tool (a
rubric or checklist that asks does it work, does it provide
instructions, etc?) might be better suited to your needs. Projects
could then be listed in order of how many or which rubric items have
been checked. I don't know who if anyone among the so-called experts
has the time to devote to assessments, but I also don't know who if
anyone among the student contributors will devote that time to peer
review. While a chunk of my life has been devoted to solving problems
with technologies, I have grown both wary and weary of technological
solutions. I hope this solution will be thoroughly tested and not just
to see if it functions but to see if it actually solves a human
problem without introducing new problems.

Best of luck,

John

On Wed, Sep 30, 2009 at 11:20 AM, Kim Rose <kim.rose@vpri.org> wrote:

[mrsteve]

To paraphrase a fellow named Alan I heard give a talk a long time ago:
"If I can do it in one click, I'll do it
If it takes me two click's I might do it
If it takes me three or more click's forget about it"

Perhaps radio buttons instead of arrows would be a better interface design.

In regards to the method of ranking/tags et al, not sure what works best, but the goals should be as stated before:
1) Encouraging kids to post their projects
2) Encouraging participation of the community in providing constructive criticism and encouragement
3)

[teefal]

Thanks for your comments and your use of the showcase. Be sure to have a look at the pending showcase tasks:

http://tracker.squeakland.org/showcase


I'll reference existing tickets below.

On Oct 5, 2009, at 11:57 PM, mrsteve wrote:

[mrsteve]

Tim,

Thanks for the rapid response.

[teefal]

Stephen,

I particularly appreciate your spanish comments for the Nicaragua and Mi Istoria Waveplace stories. I haven't yet announce that the storybooks are on the showcase, and it's very nice that the children will have their native language to read.


Also, Randy, if you're reading, thank you for your comments as well. This will mean a lot to these children.




Tim

On Oct 6, 2009, at 3:55 PM, mrsteve wrote: