Comp Copies & Reviews

Today on the spielfrieks mailing list I responded to a commonly parroted piece of ignorance: that review copies encourage reviewers to give higher reviews in the hope of getting more review copies. I pulled some stats from the RPGnet reviews database that I'm going to preserve here, because it's worthy of inclusion in a Gone Gaming article that I want to write some time. (Title: "The Ethics of Reviewing"; this is just one element.)

Out of curiosity, I did some number crunching on the review database
at RPGnet. My average rating of a product (mostly board + card games)
at RPGnet is 7.80 (out of 10). I've reviewed 166 comped (free)
products, and the average was 7.71. I've reviewed 106 non-comped
products and the average was 7.96.

I also looked at only the Substance rating, because that's the larger
part of my measure of "how good a game is", and thus I find it
slightly more meaningful. Average for a comped product was 3.70 (out
of 5) and average for a non-comped product was 3.85.

So apparently I rate free games somewhat lower than games I bought.
(It's actually a notable difference, as most ratings tend to happen in
the top half of a range, here 3-5, as I've written elsewhere in an
article on rating[1]. If you consider a two-point range being the true
breadth of a 5-point rating scale, that's a 7.5% difference.)

While I was in the database I took one step further and looked at
every review written by everyone ever. Most of them are of RPG books.
There are 1740 comped reviews at RPGnet, with an average Substance of
3.7580 and an average Style + Substance of 7.4713. There are 6184
non-comped reviews at RPGnet, with an average Substance of 3.8609 and
an average Style + Substance of 7.5867. Apparently the average RPGnet
reviewer is a little nicer than me and only rates comped books 5-6%
lower on average.

I think that 8000 data points over 10 years is enough to be
statistically significant.

I hope that the facts & actual data don't intrude too much on the
impending flamewar here, as we talk about how reviewers who *don't*
get free books artificially inflate their reviews in the hope of not
getting more free books in the future.

The parrot in question seemed pretty unimpressed and continued on with how it's a huge problem in the video game industry.

Some people just never met a fact that they didn't ignore.