|
Collective Choice: Ratings Inputs and Outputsby Shannon Appelcline Back in Trials, Triumphs & Trivialities #179, Collective Choice: Ratings, Christopher & I wrote about ratings as part of our overview of the various ways in which people can come together to make decisions. As we wrote at the time, ratings are a comparison system, whereby users say how good (or how bad) something is. In that article, we also beat up on eBay a bit for the various serious problems with their rating system. eBay did pioneer commerce ratings on the Internet, but in contrasting them with more recent ratings systems, we can see room for improvement, some of which eBay is acting on itself--and that's the subject of today's TT&T. A Question of TrustUsers face a fundamental problem when using any rating system: figuring out whether the ratings are trustworthy or not. I wrote a whole article on how we tried to make our RPGnet ratings trustworthy in Trials, Triumphs & Trivialities #196, Collective Choice: Ratings, Who Do You Trust?. Besides measuring whether users are being honest, you also have to figure out whether their ratings are in good balance with everything else in your database. But, there's an even worse problem than that: what if the rating engine itself encourages users to enter untrustworthy entries? I should step back here and say that I got to this article by way of two recent shopping experiences: one on eBay and one on Amazon zShops. In both situations I (sadly) had a bad experience. On eBay the merchant sold me an item and took my money, then revealed a full week later that he didn't actually have it in stock. Meanwhile, over on Amazon zShops I purchased a book that arrived in somewhat worse condition than was represented, including a smell of cigarettes and a peeling cover. And thus the onus fell on me to rate these situations accurately (and negatively), so that a future buyer could see the potential problems. I did that, offering up a negative rating on eBay and a neutral rating on zShops. However, choosing to enter that negative eBay rating was a real struggle, and the reason is the bilateral feedback that Christopher and I discussed in our original ratings article. To reiterate the problem: if you give a negative feedback to an eBay merchant, they're likely to give you back negative feedback, even if you did everything "right" from your side. And, the problem seems to have gotten worse since we first touched upon the topic in 2005. After I input my negative rating for the bad eBay mechant, sure enough the negative retaliation was almost instantaneous--but so was the offer to "mutually remove" the negative feedback. This is a new(er) system that eBay introduced as the problem of retaliatory feedback escalated over the last couple of years. Unfortunately, it just made things worse: beforehand customers were discouraged from leaving negative feedback by the specter of retaliatory feedback. Now they are additionally encouraged to remove negative feedback by merchants engaging in what amounts to reputation blackmail: first entering negative feedback of their own, then offering to make it go away. The overall result is that ratings on eBay have gotten very untrustworthy; you can no longer believe that a bad merchant will have any bad ratings at all. And the real irony of this all is that the problem was created by the mechanisms of the rating system itself. The good news is that as of May 1st, eBay is disallowing sellers from offering feedback on buyers. This isn't my preferred solution. I would have liked some sort of anonymous or seller-first feedback. However, this answer does broadly eliminate the problem of retaliation. There will no longer be a system in place that actively works against itself, which should increase the trustworthiness of eBay ratings as buyers no longer feel like they have to lie or keep silent. (On the downside, it also means that buyers can no longer screen "bad" sellers--though the ability to do some was always limited on eBay.) A Question of ResponseAs we've seen, there can be core problems with rating system that can undercut their very usefulness due to poor design. But, I also want to look on the flip side: how can rating systems be even more useful than usual, offering up not just information, but also added bonuses? As I said, I engaged in two bad purchases last week. The eBay purchase was just as bad as you'd expect. The seller treated the whole topic of ratings with pure gamesmanship, trying to use it to leverage himself away from the repercussions of his poor business conduct. Conversely, the Amazon zShops purchase resulted in a merchant--probably in part because he didn't have any inappropriate leverage against his customer--taking my complaints very seriously. He expressed shock when I discussed the condition of the book that I'd received, and he offered to make it good with either a refund or a replacement. The two situations make me feel that much worse about eBay's rating system and that much better about Amazon's. Whereas eBay has created a rating system that practically conspires with the sellers to disadvantage the buyers (or at least they did before May 1st), Amazon instead has created something that encourages sellers to try and better themselves. ConclusionWhen I started out this column years ago, I placed my main focus on "lessons learned." Herein I hope I offered some useful thoughts about ratings, based on real-world lessons learned as a consumer. Though I've sometimes talked about ratings on their own, it's important to remember that they're part of an electronic ecosystem. Things feed into them and out of them, and those factors can make your rating systems better or worse. Is there anything that feeds into your system and reduces the trustworthiness of your ratings, as has been the case with eBay for many years? Conversely, can your ratings have any beneficial real-world effect other than the data that they present? eBay and Amazon may offer the milestones to at least start this discussion. [ <— #206: Strategic Insights: Three Community Issues | #208: Anatomy of a Game: The Mole —> ] Miniseries ...
|
