July 9, 2019
Regardless of what you think of the various beer rating/review apps/sites like Beer Advocate, RateBeer, & Untappd they are here to stay and for better or for worse are part of the craft beer landscape. On a personal level I don’t use any of the apps, as a matter of choice despite drinking/tasting over 900 different beers in a year. Also, being in the beer business there is a conflict of interest as I inherently will have biases for and against a beer/brewery especially if I export/import them.
Having said this, while I don’t personally rate beers with any of these apps, from a business standpoint I do use the ratings as a guide for my business. They do provide guidance for me as to what new beers/breweries I should look to contact regarding exports to the markets I cover. It’s quite interesting that even the breweries I deal with that get good ratings/reviews as a whole don’t like these rating apps, but also understand their importance to their business, especially as it relates to the Beer Geek crowd which many of the breweries I deal with cater to this demographic.
As I travel outside of the US frequently for my business to meet with my import partners, visit key bars/bottle shops, learn more about the various market nuances/conditions, and to just get an overall feel for the craft beer scene I’ve noticed a cultural bias on the rating sites in particular Untappd which in my opinion (IMO) is the most used app internationally and by non-US raters/reviewers.
I’ve noticed that most Europeans will not be as “generous” in rating a beer than most Americans. By this I mean on a 5 point scale you will not see too many Europeans giving a beer a 4.0+ rating when compared to Americans. In general a beer with a 4.0+ rating is on the “exceptional end” especially when you are rated 4.1+. As I look at Untappd ratings quite a bit for my export business and also looking to import some international beers to the US, I’ve noticed a general trend where you don’t see to many non-US breweries with an overall brewery score of 4.0+ with the exception of the top Belgian gueze/lambic breweries. These top international(primarily European) breweries that I am talking about are considered to be the Top 60-80 craft breweries in Europe and not many of them have 4.0+ Untappd ratings. Some in fact, by American standards, have a “pedestrian” 3.6 overall brewery rating.
While I have noticed this “trend” for some time, I’ve never really questioned or delved into the reasons or implications of this until I had a booth at a beer festival in Flanders back in November 2018 where I came “face to face” with this “country/cultural ratings bias”.
I was participating at a consumer beer festival in Belgium where I had a booth and was able to engage with primarily Flemish consumers. This was a token based festival where a consumer would pay for ~15cl beer sample with X # of tokens. Any case, at the festival I would look at the Untappd check ins and ratings for the beers I had at the festival and similarly noticed a “downward ratings” trend even for the highly rated beers I had which were 4.0+. I had a double dry hopped beer I export to Europe with an Untappd rating of 4.4, so on the highly exceptional end. I managed to chat with dozens of consumers at this festival and would ask them how they liked the beers to see which beers they liked and what they thought about the beers I had. One particular chat was enlightening when I was talking to two attendees who really liked the double dry hopped beer with a 4.4 rating they sampled. They were in fact raving about it and that it was definitely much better than the regular version of this beer which they also tried and liked, but definitely preferred the double dry hopped version and understood why it got a higher rating than the regular version which has a 4.1 Untappd rating. They proceeded to tell me that they would give the double dry hopped version a very good rating on Untappd. Thus, I was expecting at least a 4.0+ rating, but saw both of them score the beer a 3.75. I didn’t question why they rated it “just a 3.75”, but was honestly surprised as this is just above average by “American rating standards”. Similarly, looking at the other check ins of the other beers I had at this festival and they similarly had a “downward trend” in ratings despite positive or even overly positive comments noted on the beers.
Several weeks later, I had a chat with several Belgian and Dutch brewers I know of regarding my experience at this consumer beer festival and asked them on their opinion and they similarly agreed and noticed that Europeans tend to give lower overall ratings. The next question is why this is and what theories or explanations to this phenomena. Some have suggested that the European educational system is more stringent in giving “top scores” so culturally Europeans are raised to be more critical in giving out ratings and thus not as generous than the average American. Also, culturally, nothing is deemed “perfect” by society so the top score or ranking is rarely given. They mentioned to me that friends of theirs never or rarely give a beer a top score of five as nothing is “perfect” and similarly rarely rate beers 4.0+ and ratings this high are reserved for less than one percent of beers they try. Interestingly, if they think a beer is “just average” they rate it a two or in the “two range”. By “American standards” a “two range” rating is a below average beer and in general an “average beer” would be rated a three.
What I have observed is quite anecdotal, so I wanted to see if I could find some quantitative evidence for this “country/cultural bias” on Untappd ratings. As of March 29, 2019 when I checked on Untappd’s 50 highest ranked breweries by country I found:
Australia: only one brewery rated 4.0+ out of 605 breweries in Australia based on 2018 brewery count according to Craftbeerreviewer.com(http://craftbeerreviewer.com/the-brewery-list/), this translates to just 0.17% of breweries with an “exceptional” rating of 4.0+
Austria: highest rated brewery 3.8 out of 273 in 2017 (Brewers of Europe website https://brewersofeurope.org/site/countries/figures.php?doc_id=689), thus no breweries considered “exceptional” with a rating of 4.0+
Belgium: seven breweries rated 4.0+ out of 271 breweries in Belgium in 2017 (Brewers of Europe website) or 2.6% with and “exceptional” rating. Note, six of the seven breweries are gueze/lambic breweries with only Sint Sixtus Abdij/Westvleteren (4.42) the only non gueze/lambic brewery. Not even the Trappist breweries from Chimay, Westmalle, or Orval are rated above 4.0 which does not make any sense. This translates to 2.6% of Belgian breweries with an “exceptional” rating.
Canada: 12 breweries rated 4.0+ out of a total 817 breweries in 2017 (BeerCanada) or 1.47% of breweries with “exceptional” rating. https://industry.beercanada.com/statistics
Czech Republic: highest rated brewery 3.9 out of 450 (Brewers of Europe website)
Denmark: only one brewery rated 4.0+ out of 197 (Brewers of Europe website) or 0.51% “exceptional”
France: highest rated brewery 3.9 out of 1100 (Brewers of Europe website)
Germany had one brewery rated 4.0+ out of 1500 breweries in 2019 according to the German Brewer’s Assoc website or 0.07% of German breweries have an “exceptional” rating https://www.reinheitsgebot.de/en/home/german-brewers-association/
Italy: highest rated brewery 3.9 out of 868 (Brewers of Europe website)
Japan: highest rated brewery 3.9 out of 174 (USDA Foreign Agricultural Service GAIN report 8/31/18)
Netherlands: one brewery rated 4.0+ and it’s a “gueze/lambic”/wild/mixed fermentation brewery out of 500 (Brewers of Europe website) or 0.2% exceptional
Norway: highest rated brewery 3.8 out of 136 (Brewers of Europe website)
Poland: highest rated brewery 3.8 out of 210 (Brewers of Europe website)
Spain: one brewery rated 4.0+ out of 521 (Brewers of Europe website) or 0.19% exceptional
Sweden: one brewery rated 4.0+ out of 338 (Brewers of Europe website) or 0.16% exceptional
Switzerland: one brewery rated 4.0+ out of 869 (Brewers of Europe website) or 0.12% exceptional
UK: only four breweries rated 4.0+ out of 2430 (Brewers of Europe website) or 0.16% exceptional
The one “aberration” was in Brazil which had 22 breweries rated 4.0+ out of their top 50 brewery list. The latest statistics I could find for the number of breweries in Brazil was 679 in 2017 which translates to 3.2% of Brazilian breweries having an “exceptional” Untappd rating of 4.0+.
Outside of Belgium (2.6%), no European country had over 1% of total breweries with a rating of 4.0+ and only Brazil with 3.2% and Canada with 1.5% were over 1% of total breweries with an “exceptional” rating. For Belgium the fact that that they have seven breweries 4.0+ is probably the result of these top gueze/lambic breweries and of course Westvleteren being a “bucket list beer” for American Beer Geeks so they have a large amount of American reviews which skew their ratings higher than other Belgian beers. It is amazing that Austria, Czech Republic, France, Italy, & Poland had no breweries 4+, Denmark, Germany, Spain, & Switzerland had only one brewery each, and the UK only four breweries.
Next, I wanted to see how this compares to US breweries. As I could not easily quantify all the US breweries rated 4.0+ on Untappd, I filtered select US states for their Top 50 ratings (as of June 4, 2019) and using the 2018 US Brewers Association data for the number of total breweries in each state to get the percentage of “exceptional” breweries with a 4.0+ rating to come up with the below (https://www.brewersassociation.org/statistics/by-state/ ) findings:
Alaska: one brewery 4.0+ out of 42 or 2.4% “exceptional”
Arizona: seven out of 117 or 6.0%
California: 32 out of 841 or 3.8%
Colorado: 10 out of 396 or 2.5%
Connecticut: nine out of 87 or 10.3%
Florida: 20 out of 285 or 7.0%
Georgia: four out of 82 or 4.9%
Illinois: 16 out of 229 or 7.0%
Indiana: five out of 181 or 2.8%
Iowa: four out of 94 or 4.3%
Kansas: one out of 47 or 2.1%
Louisiana: one out of 39 or 2.6%
Maryland: five out of 94 or 5.3%
Maine: six out of 119 or 5.0%
Massachusetts: nine out of 155 or 5.8%
Michigan: seven out of 357 or 2.0%
Minnesota: four out of 178 or 2.2%
Missouri: eight out of 112 or 7.1%
New Hampshire: six out of 81 or 7.4%
New Jersey: six out of 109 or 5.5%
New York: 25 out of 386 or 6.5%
North Carolina: 11 out of 291 or 3.8%
Ohio: nine out of 290 or 3.1%
Oregon: seven out of 284 or 2.5%
Pennsylvania: 13 out of 354 or 3.7%
Rhode Island: four out of 26 or 15.4%
South Carolina: five out of 82 or 6.1%
Tennessee: three out of 99 or 3.0%
Texas: ten out of 283 or 3.5%
Vermont: 14 out of 66 or 21.2%
Virginia: eight out of 236 or 3.4%
Washington: seven out of 394 or 1.8%
Wisconsin: seven out of 190 or 3.7%
The US ratings are obviously statistically higher than the international countries when comparing the total number of 4+ breweries and you have 14 states(out of 33 states) with percentages of 5% or higher of “exceptional” breweries. You have some states like Rhode Island (15.4%) and Vermont (21.2%) that were off the charts.
While the above anecdotal evidence of a country or cultural ratings bias is probably correct, determining or quantifying really how much it impacts scores would be the next step by parsing the data by ratings by country could lead to some interesting insights and findings. As I do not have access to Untappd’s raw data, I am taking an educated guess is postulating that breweries with a mainly European ratings base would probably have about 0.3 to 0.4 higher ratings than what their current rating reflects. For example, if a Dutch brewery has a 3.6 rating and adjusting for this bias they would really be a 3.9 to 4.0 from a US perspective. Conversely, for US breweries if reviewed from a European perspective, you would deduct 0.3 to 0.4, so a US brewery that is a 3.8 would be 3.4 to 3.5 on an European scale.
Similar to these country/cultural ratings biases, other biases I’ve noticed are based on beer style where the two most pronounced ones to mention are that raters tend to skew negative or lower for pils/lager styles and conversely tend to skew more positive or higher for stouts, especially barrel aged stouts. It is rare to find a pils/lager style 4.0+ and you probably see the highest percentage of stouts rated in the rarified 4.4+ range. Maybe in another post I’ll delve into this if someone has not already done so.
So what are the implications of these country/cultural & other biases in Untappd ratings? I suppose the first thing is being aware of them, that they exist, and thus understanding the caveats to the scores you see. Then, depending on how much you use/rely on them for purchase decisions to be able to “interpret” these ratings properly due to inherent biases of the reviews/rater. The take home message is that while these ratings are “quantitative”, there is a country/cultural subjectivity of what is considered or deemed a “high”, “good”, “outstanding”, “average”, “poor” ratings/score.
The purpose of this post is to at least make those who use these rating apps aware of this “country/cultural bias” and the breweries which have a heavy European audience will probably have “lower ratings” than breweries which have a primarily US audience and “vice versa” and to take this into account in your decision making. It is acknowledging this subjectivity that IMO is important as the impact of high ratings early in a new brewery’s genesis can be vitally important to their near term success, “trendiness/hype factor”, and can “make or break” a new brewery.
Beverage ratings & reviews have been around for over 40 years in print form, especially in the wine & spirits sector, so rating beer is nothing new or novel. However, the ubiquity of the internet and more importantly, the significant impact of social media in our daily lives has magnified both the speed and broader reach of these beer ratings/reviews from just a local or regional impact to a worldwide audience in mere minutes. Also, the relatively new phenomena of drinking beer as a “sport” where earning a badge or reaching a level has added to the “gamification” of drinking beer which taps into (pun intended) our inherent need to compete with one another and out do one another.
The intent of this post is not to dissuade anyone from rating or checking in a beer, as millions of people world-wide enjoy these rating apps/sites, the community/camaraderie associated with them, and utility/functionality in using these apps, but to just be cognizant that a “3 just might not mean a 3” based on your perspective.