Monday, March 18, 2013

Beer competitions

The subject of judging at beer competitions came up on that twitter again.  There are valid concerns about the credibility of these things which get raised from time to time, as well as the broader question of "what's the point of them anyway".  First some specifics:

How are beers tasted blind?

Speaking from personal experience as one of the organisers of the SIBA North Beer festival / competition, "blinding" the judges (as far as possible - and I'll come back to that) is something that's taken very seriously. While the actual organisation of the competition is down to SIBA's professional staff, the logistics are the joint concern of the "central powers" and the regions.  The way it works is this: A load of judges are  assembled in a big room, and a load of beer in another one. A further area is reserved for beer handovers. One bunch of people (the cellar team) draw the beer from the (easily identifiable) containers, take it out of the cellar, and hand it over to the volunteer runners who carry the jugs (labeled with a code) to the judging tables.  The SIBA central staff (behind a screen) hold the key and can (later) figure out which beer is which.

Now, short of active cheating, or marked incompetence, this system ensures that no judge can know which container the beer they're tasting came from. 

So you have a table full of judges sniffing and tasting beers, holding them up to the light, pulling faces, passing comments, holding forth on what makes a great beer, asking the self-appointed experts what they think, trying to sway each other, etc, before marking down their scores for a bunch of predetermined criteria. Munge all these scores together and you can pick the highest scoring in each category, which can go on to be judged in another round, and so-on, until you get the top 3, and award them Gold, Silver and Bronze in their classes.

All the gold winners can be judged against each other (tough job, depends how you do it) and a supreme champ and a couple of runners up can be decided upon. 

That's one way of running a blind tasting beer competition.

What's wrong with this picture?

In a sense, everything.  Firstly, the judges. Someone who drinks a lot of beer, and who has a fair idea of what the beers entered might be, can probably recognise some beers.  So they're not 100% blind.  Of course we can also include (and we do) some people who don't necessarily drink a lot of beer.  Which you might think would be a disqualification for judging.  But there you go. 

Blinding isn't the big problem that some people seem to think it is.  I don't think cheating is a significant problem, and I believe most of the judges would indeed score a great beer higher than a merely fair one that they happen to be familiar with.

I don't know where you, dear reader, stand on CAMRA,  but it's worth mentioning that a fair percentage of our judges are selected from among CAMRA activists.   Many of these will be trained tasters with a lot of experience,  highly qualified to judge at this sort of thing.  Some will be utter gobshites.  Same goes for judges from the licensed trade.  You could argue that the same could be said for the brewers who judge.

And of course, there's a tendency (in all of us, I suspect) to prefer the things we're used to.  So there's probably some bias in the judges toward the kind of beers that they're more comfortable with - which gives us a tendency to conservatism.  Far-out beers need not apply? Beers in less popular styles might not do as well as an aficionado might think they should.   It's a beauty contest judged by fallible humans who bring their own prejudices to the table, not an objective test of beer greatness.

Consider also (he said, donnishly) the dynamics of small groups.  There's always some tit who wants to "lead" the group. And sadly, perhaps, there are those happy to be led. So instead of a hundred independent judges we might end up with a few loudmouths commanding votes disproportionate to their expertise (but in line with their sense of self-worth).  N.B.  I say "might", I have no personal experience of this happening in a beer competition. 

Why do we need so many judges? There are only so many respected beer judges in the world and many of them would at least want expenses to swig and sniff for a day for us.  But a judge can taste (and swallow) only so many beers before falling over.  So a lot of beers need a lot of judges, which means the quality of the judging pool isn't necessarily as high as you'd like. We have some great judges, of course, and a number who are less great. 

And then there's the judging criteria.  We score beer on things like "appearance" (including clarity?) and "sale-ablity".  What about hazy beers?  What would it mean to have a beer scoring well on everything except  "sale-ablity" - is this a trap placed in the way of stronger beers?

The award system itself is problematic - are only three beers (the medal winners) any good?  Are the others, which didn't make the cut, therefore crap and not fit to drink?

Who is the competition for?

We love awards.  Validation. Confirmation that it's not just us who think our beer is lovely - See my awards!  I'm a real talent!  My beer is great! That other beer isn't.

When a brewer wins an award they tell the public.  There's an idea that it'll help us sell beer.  Our customers must surely know that because it won an award it's good beer - better than other beers they could buy. We might get a bit of press for it.  Drinkers might have heard of it before they see it on the bar, or on the shelf, so retailer risk is reduced.

Drinkers can order a pint (or buy a bottle) of an award-winning beer secure in the knowledge that it's going to be better than mediocre, or even, really good.

Or is it just (and particularly for a trade organisation like SIBA) a mutual back-slapping exercise.   Brewers telling other brewers (who make the kinds of beers that the herd of brewers like) that the herd of brewers liked the beer.  A beery circle jerk.  

What's to be done?

A beer competition has to be credible.  It has to offer valuable information to the drinker.  Without that it's no use to them.  And if no use to the drinker, what real use (apart from the warm feeling we get when our fellows give us a pat) is it to the brewer?

I'd propose 3 main changes:

A reputable, external chief judge (or several).  We need a name that the drinker respects.  And not just as a figurehead.  The competition needs to be reviewed and this person should be involved with that process 

Judging criteria should be reviewed and published for debate.  Get rid of sale-ability for a start - that's nonsense.  How can a good beer be not sale-able?  Would it depend on the price? Bollocks.

The awards structure should be revised. It's not bloody Highlander ("There can be only one").  And it's not the Olympics (what does bronze mean? Good, but not really all that good?). All the beers which are excellent should be recognised.  Sure, pick one to be champ, but make it clear that this one is the judges pick of the excellent.

If you've read this far, thank you. If not, f**k you.


Dave Bailey said...

I think you'll find "Bronze" means "bloody great and only a smidgin less great that that one that got Gold. Oh, and significantly better than all the others that got nowt"

StringersBeer said...

Sure, gap between 1 & 3 = smidgen whereas 3 & 4 = great gaping void.

Cooking Lager said...

I think before I started reading beer blogs I wouldn’t ask that many questions of a beer award. I’d simply assume it won an award it must be good and maybe even try it as a preference for trying something else. I think I am a little more sceptical these days regarding from whom the award is being given as to whether it is a recommendation. No matter.

As you guys enter these competitions I’d guess it a fair assumption you do so for the purpose of winning so it must in some way be a worthwhile thing to win. Afterwards you have to reconcile either winning or losing so I guess the notion that the game is fair, the rules are clear, the rules are applied equally to all are significantly important. You might also wish the rules somewhat favoured what you see as your own strengths but might have to accept others determine what the game is.

The desire to alter the game, to add or remove criterion is I think the desire to play a different game, the desire to improve the judging a desire to see the result as fairer. Even so, wanting to win is why you enter in the first place.

StringersBeer said...

I'm not saying that beers that win these competitions aren't good. They are. So, particularly for those who don't know a lot about beer, they're a fairly reliable indicator of a kind of quality.

As it happens, we enter beers which we believe to be good, without holding out any particular hope for them winning anything (although we have picked up regional bronzes and silvers, and been jolly pleased). Entering something is the easiest thing a member can do to support the competition, increase the value to the winners and expose the judges to a wider range of beers.

The problems start when we see beers which we know to be excellent getting no recognition in the competition. This is when the dark mutterings start. The point I'm making (I hope) is it doesn't require bad behaviour on the part of the organisers or competitors, or wickedness on the part of the judges, for the results not to reflect the high quality of the entries.

The competition is essentially a service to the members. But for it to be that, it needs to provide valuable information to the public.

Brother Logic said...

I have no problem with non-expert judges passing their opinions, I don't think you really need to have drunk any amount of beer to decide which one you liked the best.

Scoring's a huge problem since it requires judges to calibrate with each other - a better system (though more judges would be needed) would be to get judges to rank rather than score since this removes the need for calibration.

In addition, if there is discussion amongst the judges then the competition is invalid in my opinion. You can't mix discussion with a scoring system, it's just statistically wrong and will lead to bias in the results.

StringersBeer said...

Bro Logic, I'm with you there. Of course it's trivial to convert judges uncalibrated individual scores (for a number of beers) to rankings. And there's the issue of outliers - we've seen some extreme cases of this - basic statistical hygiene says that these should be discarded.

FyneJamie said...

Interesting article. I guess you have to expect any process that's about people making judgements to have some variability in it - so it's never going to be a suprise to see some great beers not winning in any individual competition. I always think you have to look at the awards process as a totality - knowing that you will win some and lose some, but if your beers are good (and you take the time to enter the competitions) you will win more.

The element that you didn't discuss was the number of classes and the guidelines for the classes that SIBA and CAMRA both judge against. By having just a few very broad classes, I think there is a risk that the awards will too often hold up beers that are relatively middle of the road as being the best, as this will be the one judges can coalesce round. Personally I think it might be interesting to adopt the BJCP guidelines or a subset of them, in order to encourage excellence in brewing lots of different styles. It might really help to open the market up, and to educate consumers.

StringersBeer said...

But of course, picking the best isn't necessarily what we're most interested in - we might choose to identify all those of some certain standard...

FyneJamie said...
This comment has been removed by the author.
StringersBeer said...

FyneJamie, you're right of course - the classes should be considered in any review.