Ranking crowdsourced data with curves

I’m a big fan of user review sites.  I’ve been using TripAdvisor since at least 2002 to help plan my journeys, and Yelp is my new favourite site since moving to the US.

Crowdsourcing information is usually a pretty good way of doing things.  There’s been plenty of research which has shown that if you get a crowd of people together and have them, for instance, guess the total number of jelly beans in a jar, the average guess will be pretty close to the real number.  Translating this to something usable in everyday life, I’ve seen people have a lot of success with Amazon’s Mechanical Turk for aiding research, since you can use the sheer weight of numbers to smooth the data.

This is great for something tangible and objective.  However, as soon as you start throwing subjective data into the mix, results get a little skewed.

In particular I’ve noticed lately I can’t quite trust online reviews (from a wide variety of users) the way I used to.  I’m not sure why things have gotten so skewed — perhaps I’m more discerning now, or maybe there are more outliers.  Either way, I haven’t trusted the average rating on sites like Amazon, Yelp and TripAdvisor for a couple of years now.  Instead I read a sample of reviews and then go straight to the upper and lower bounds and try to get a sense of why people are voting in a particular fashion.  Is the product/place being reviewed being unfairly penalized or rewarded? (for example, a hotel might get a lot of 1 star reviews for high parking fees, but are great otherwise.  A restaurant might get a lot of 5 star reviews because they’re cheap and have a nice ambiance, even though the food stinks.)

One method I’ve found for making a better decision is to look at the shape of the score curve.  For example, here are the scores for the top 5 TripAdvisor hotels in San Francisco at the moment, order by TripAdvisor by their average score:

1

2

3

4

5

There aren’t any that really stick out there as being obviously different, but you can see that the fourth one gets a far higher ratio of 5 star ratings to 4 star ratings than the others. These differences become more pronounced the further down the overall list of hotels you go. Hotel A is rated as a better hotel than Hotel B which is ranked (according to its average) after it:

Hotel A:

Hotel B:

Whenever I see a curve difference like this though, I always go for the latter when booking.  Since I changed tactics, I have been having great hotel experiences.  To give this a quantifiable score to compare, I tried out the following formula:

If ((x star votes) – (x-1 star votes)) > 0 then

y = x

else

y = 5 – (x – 1)

(y*(5votes – 4votes) + y*(4votes – 3votes) + y*(3votes – 2votes) + y*(2votes – 1votes) + 1votes) / total votes

Using a formula such as this the new overall scores become:

(260 + 96 – 9 + 8 + 1) / 128 = 2.78125
(600 + 108 + 6 + 8 + 2) / 206 = 3.51456
(1550 + 548 + 54 + 18 + 5) / 699 = 3.11159
(425 + 28 + 15 + 0 + 1) / 119 = 3.94117
(1260 + 948 + 60 + 16 + 13) / 883 = 2.60136

Which as you can see results in a much better looking ranking for the curves:

4

2

3

1

5

If we apply the formula to the hotels A and B, we see the difference becomes more pronounced:

(-26 + 244 + 15 + 22 + 6) / 185 = 1.41081

(250 + 236 – 9 + 46 + 10) / 301 = 1.77076

Introducing a new overall score would help people pick better hotels and for the hotel owners to strive for higher ratings.  I’m also a big fan of the trending data that Yelp has added recently, using Patxi’s as an example:

Coming from someone who wrote his thesis on qualitative user feedback, this has been really interesting for me to look at how you can properly interpret large amounts of quantitative data involving subjective scores.

Update November 7 2008: Thanks to Eric Liu for pointing out some weighting issues depending on vote numbers. We’re brainstorming some new algorithms to account for these situations.

In the meantime, anyone from the myriad of Netflix people who have stopped by, feel free to contact me! tim@<this domain>.

American food is killing me

I’m completely addicted now.  In-n-out.  Krispy Kreme.  Sibby’s Cupcakes.  Fred steaks.  And tonight’s indulgence?  Chicago style deep-dish pizza from Patxi’s.

We ordered a 12″, which weighs nearly 2kg (it is 2″ deep).  It takes 30 minutes to bake (compared to the usual 6 minutes or so for a thin crust pizza).  The pizza actually consists of two layers.  Ingredients are placed on a bottom layer of dough, another layer is added on top of that, and then the sauce is placed on top.  We actually ordered the “low fat” version originally, but the kindly gentleman who took our order swiftly talked us out of that.

Paxti's original pizza

Delicious.

Does anyone actually notice a website being ‘free’?

I know that ‘freemium‘ is a popular business model at the moment, but do everyday web users really notice this when they’re using a site?

Personally I always assume a site is free and am surprised/annoyed when it isn’t, unless it offers something very compelling or a freemium model (the best, in my opinion, is Flickr).  However a lot of sites put “free” in large type (including Trovix!) on their landing page.  The only rationales I can think of (beyond the rare case that most other examples of your product are paid) split into two camps:

1) If you require an account to use the site, some people might think they will be asked for credit card details during the sign up.  But how prevalent is this concern?  It’s never come up in any focus groups or user testing.  When directly asked, the people in my study groups were unanimous in the assumption it would be free.

2) It’s used as an attention grabbing bit of text that is intended to make the user feel like they are getting something (which should be a paid service) for free.  This just feels tacky, and depending on how it is done, can detract from the brand.

I was struck by Mint’s use of this tonight, which prompted this post:

Mint\'s logo with \'free\' in it

Perhaps it is a cultural thing — I have seen many products that label themselves as ‘free’ in a scammy way.  Regardless, unless everyone else is charging (or even if they are, like when Yahoo and Google introduced real time stock prices), it seems an unnecessary feature to draw attention to.

Why isn’t anyone talking about Google’s spam problem?

Sometimes when I’m searching for a particular product or service, the most frustrating thing is the way Google brings up mostly things to purchase.  For example, recently I knew that a direct flight between Sydney and New York used to exist, and wanted to find out some information about it.

These are the top results:

  1. Cheap Flights to New York , USA from Sydney

    Compare cheap flights to New York , USA from Sydney , Australia with Airfares Flights.
    www.airfaresflights.com.au/flights/Sydney/NewYork-NYC/ – 154k – CachedSimilar pagesNote this

  2. Cheap Flights to New York JFK , USA from Sydney

    Search Sydney to New York JFK Flights. Search More Options Below (searches may not include deals above). Sydney to New York JFK Air Travel
    www.airfaresflights.com.au/flights/Sydney/New_York-JFK/ – 50k – CachedSimilar pagesNote this

  3. Flights New+york Sydney

    Flight from New+york to Sydney from 453 euros. Search every airline including low cost. Return One way Direct flights only Main airports only
    www.edreams.com/flights/new+yorksydney/ – 21k – CachedSimilar pagesNote this

  4. Flights Sydney: Cheap flights to Sydney, eDreams

    Flights New YorkSydney, 1 November 2008, 453€, United Airlines, John F Kennedy Intl Airport, ….. Direct flights only Main airports only. About eDreams
    www.edreams.com/flights/flights-to-sydney/ – 68k – CachedSimilar pagesNote this

To cut through this I like to use Google Blog Search, where I can usually find some helpful blogger who is talking about the concept I’m interested in.  However sometimes, like this occasion where I’m essentially searching for old news, Google Blog Search fails to remove the cruft, as it tends to give an inordinate amount of weight to more recent posts:

European and Asian Stocks Fall Sharply

8 hours ago by Jack Kelly
Total, the French oil company, slipped 4.5 percent on heavy volume, as oil futures for November delivery on the New York Mercantile Exchange fell as much as $3.92 to $89.96 a barrel on the New York Mercantile Exchange.
CompliancEX – http://compliancex.typepad.com/compliancex/
[ More results from CompliancEX ]

Stocks Fall Sharply on Credit Concerns

5 hours ago by chrisy58
New York Times. http://www.nytimes.com/2008/10/07/business/07markets.html?_r=1&hp&oref=slogin. October 6, 2008. The selling on Wall Street began at the opening bell on Monday and only intensified as the morning went on.
Chrisy58’s Weblog – http://chrisy58.wordpress.com/

Credit Crisis Drives Stocks Down Sharply2 hours ago by Ricardo Valenzuela
Crude oil was trading just over $89 a barrel in New York after 2 pm. President Bush made an unscheduled stop on Monday morning to speak about the crisis with owners of small businesses in San Antonio — and the television cameras that
INTERMEX POWER – http://intermexfreemarket.blogspot.com/

As such, one of my most often used Google sites is Google Groups — its long-term archiving and generally high quality results combined with a more traditional Google search algorithm mean I can usually find very specific discussions I’m looking for.  For example, in the past I’ve used it for finding out people’s experience with getting a green card, sourcing older tech support, or travel experiences.  Lately though, Google Groups has been completely broken.  Here are the results for my search:

Direct Flights Nassau To New York Group: obs27rruhuvtene

auwwb…@dipnoi.az.pl obs27rruhuvtene *Direct Flights Nassau To New
York* <http://sebj2o8a3l.airlinetickets24.info/?i=30091017&data=airline+ticket>
Direct Flights Nassau To New York Flights New York Dublin <http://groups.google.
com/group/obs27rruhuvtene/web/flightsnewyork-dublin>, Flights Sydney To New York
Sep 16 by auwwb…@dipnoi.az.pl

Direct Flights New York Group: skfea3bmsss

mvzdilis…@anosmia.az.pl skfea3bmsss *Direct Flights New York* <http://0vcz7v.
airlinetickets24.info/?i=30091020&data=airline+ticket> Direct Flights New York
Larnaca cheap flights, Wholesale cheap airline tickets, Cheap flights from
cardiff to majorca, Airline tickets bid, Cheap flights from brisbane to sydney,
Sep 19 by mvzdilis…@anosmia.az.pl

Cheap Flights From Sydney To Aukland Group: Data Access rodeo1a

Amsterdam 59.00 New York 179.00 Bangkok 389.00 Caribbean 299.00 Sydney 639.00
Flights Sydney Auckland Compare and book Cheap FlightFlights Sydney Auckland
Sydney.Cheap Flights to Sydney Australia from UK Airports TRAVELBAGTravelbag
provides cheap flights to Sydney flights from London direct flights Sydney.to

Apr 24 by susannabatchelorb8b…@gmail.com

It seems like Google has given access to creating groups to anyone with a Google account.  This, combined with the long broken Google CAPTCHA means it is getting completely clogged up.  I went 250 pages in (which in itself surprised me, as you can only go 100 pages using normal Google search) and there it was still.

So I tried a few other searches I’ve done lately.

Roomba battery? Broken.

Olympic games? Partly broken.

Travel to Ghent? Broken.

This one probably isn’t fair.

Anyway, there certainly is a lot of crap in there.

I’ve been noticing how useless Groups has been for several months now and assumed Google would do the usual thing and fix it up pretty fast.  I just checked and the last mention of Groups on the official Google blog was in January 2007 when this feature was taken out of beta, meaning it’s a problem wasn’t fixed in the 18 months it was in beta.

I heard one of the reasons why PicasaWeb limits a user’s space (as opposed to Flickr’s unlimited paid service, a sore point I won’t discuss now) is to “prevent spam” — if they can choose to fundamentally cripple a product in their fight against spam, why not fix one that is broken by spam?

Update: Apparently even more spam issues at hand with Google Groups.