Post image for Bible Lessons

Bible Lessons

by Oliver Klimek on February 22, 2010

Thoughts on Jim Murray’s Whisky Bible 2010

At the Finest Spirits Festival in Munich last weekend I had the chance to have a few words with Jim Murray who had been placed in a rather forlorn corner of the basement. I took the chance and bought one of his Whisky Bibles which he was kind enough to sign for me. Unfortunatlely I did not have the time to attend his “Jim Murray Gives it to You Straight” masterclass scheduled very late at night.

So far I had only read second-hand reports about his ratings for some particular whiskies, most notably his repeated choice for an Ardbeg as Scotch Whisky of the Year. This move had provoked a lot of criticism in whisky world – especially on the internet in blog posts, forum threads and Twitter tweets – culminating in accusations that Jim Murray was on Ardbeg’s or LVMH’s payroll. I have to admit that I wasn’t immune to such thoughts myself. But I accepted Jim’s blog comments denying any affiliation and that the reason was merely that he liked Ardbeg very much.

Now that I have had a thorough look through the book, I am assured that his explanation was genuine. There are so many other whiskies with extremely high scores from many other distilleries that it is hard to believe that any money was involved in this.

The High Score Mystery

But this plethora of high scores has left me very thoughtful nevertheless. So many whiskies have extremely high scores – including some quite simple ones – that I am having a have a hard time to align my own impressions with those of Jim Murray. I know I am not alone with this feeling because Jim tackles this issue in his book where he admits being critizied for marking too highly. He defends the fact that 50 out of 950 score 95 or better, stating that this percentage of about 5% “sounds about right” to him.

I’m afraid I have to disagree on this. If you take culinary guides like Michelin or Gault Millau that rate restaurants, 95+ scores for whisky would be comparable to the top restaurants with three Michelin stars or a Gault Millau rating of 19 or 19.5 out of 20. Now would you expect 5% of all restaurants to be temples of ultimate culinary delight? Certainly not. Paris or London have thousands of restaurants, but only a handful have top scores. Most are not even considered to be included in these guides because they are just not good enough. The rate of top restaurants will probably be around one in a thousand, if at all. The underlying statistical law is the Gaussian distribution with a big bulge at the median value and very flat tails at both extremes.

The Whisky Bible takes a holistic approach and wants to list as many whiskies as possible. So we can expect a statisically relevant cross-section. 5% of all whiskies in absolute top positions would mean that the overall quality of whisky is extremely much higher than the quality of restaurant food. Is this really true?

The Dangers of the Compound Score

I think the answer to this mystery can be found by looking closer at Jim Murray’s approach to rating whisky. Jim’s ratings are diveded in four sub-scores for nose, palate, finish and balance. Each component is scored separately on a 25 point scale and then added together. His marks for each of the components are 20+ most of the times. So whenever a whisky is at least “decent” in Jim’s opinon, he is left with only 6 marks for each component.

This is effectively a five star rating which he strongly disapproves as a rating system for whisky in the introduction to the book as being too inaccurate, and rightly so. This is indirectly proven true by the introduction of half points because with only full points there would be too litte possibility to fine-tune a score.

And there are three more important things to consider with Jim Murray’s compound ratings:

  1. The “Balance” score is not independent from the other three. In the author’s own words: “For a whisky to work well on the nose and palate, it should not be too one-sided in its character.”If this is to be taken seriously, an unbalanced whisky will never be able to receive top marks for nose and palate (and certainly also finish). So in a way, this fourth score only pretends to give additional information.
  2. Because it’s a human who does the scoring and not a gas chromatograph, there is always a margin of error involved. Sampled on another day, any whisky might get a point more or less. Four scores mean four margins of error. 21/19/22/22 on day A could well give 20/18/21/21 on day B or 22/20/23/23 on day C. The resulting score would be anywhere between 80 and 88.
  3. All four sub-scores are weighted equally. Does this hold true for everyone? I for one tend to weigh the nose far less than palate and finish. But in my experience, more whiskies have a better nose than palate than vice versa. And dow does the calculation take into account that the balance score is connected with the others?

Where is the Calibration mark?

The “continuous” 100 point rating system I adopted from the Malt Maniacs and that was also used by Michael Jackson to my knowledge (correct me if I’m wrong) uses 75 points as a sort of calibration mark. This would be a run-of-the-mill whisky that you can’t really call bad but that has nothing to write home about either.  When scoring a whisky using this system, the final score will be an assessment on how far away it is from that generic 75 point dram.

For a compound score to work well, you would need a calibration mark for each of the sub-scores. I would never accuse Jim Murray of not having some sort of neutral point for his scoring. But I tend to think that it is placed rather at a sub-score of 20 than at 18.75 (75/4). This would explain why his ratings are often significantly higher than other people’s.

But perhaps Jim’s ratings just reflect that he is generous with his scores because he likes whisky in general so much. Which is not a bad thing at all, mind you.

Does 95 Alwyas Equal 95?

But there is yet another thing that struck me browsing the Whisky Bible. A simple Irish blend like Jameson receives a score of 95 and – even more astonishing – the Swissky Exkusiv-Abfüllung from Switzerland gets 94.

What would Jim Murray say if presented with a head-to-head sampling of a Jameson and lets say an Ardbeg Airigh nam Beist that he scored 95 points as well. Or how about the Swissky and a Glenfarclas 17 (93 points)?

I can’t help thinking that these scores are not comparable. Coming back to the restaurant guide example, I guess giving the Jameson 95 points means something like “Here you can get the best Fish & Chips in town” while the Ardbeg score compares with the two star restaurant serving truffled lobster mousse with scallops.

Is this Book any Good Then?

Yes! The Whisky Bible is a great collection of tasting notes and scores for thousands of whiskies, and you really have to love Jim Murray’s writing style.

But you should be aware, that these are just one person’s opinions, worth as much or as little as any other’s. Just because a writer has a big name, it does not mean that you will like what he likes. I have found out that Jim Murray’s ratings have a very erratic correlation with my own, but this does not keep me from enjoying his book.



{ 11 comments… read them below or add one }

gal granov February 22, 2010 at 8:10 pm

Oliver,
a truly great post. very well written and on the money!

i agree fully. i do rely on the bible as a reference but not on the grades.

chapeau!

Reply

Elvar February 23, 2010 at 9:44 am

Very useful thoughts. The remark about calibration mark makes sense. For instance I never understood how Michael Jackson could rate a lot of small whiskies around 70 when I found them to be more like a 50 in comparison. Even though I agreed with his taste most of the time I found this confusing. Why doesnt he just start at 60 and use 0 to 40. It probably would have made lot of producers very unhappy. As for Jim Murray, invaluable as his book is for the wealth of informations the scoring system is very misleading for the normal drinker. Its not until you know which ones can really compare that its useful.
Elvar, Reykjavik

Reply

Bernhard Schäfer February 23, 2010 at 8:31 pm

And never forget, the marks are just the opinion of one person. This can never resemble anything near objective.
But his book is much more, than his “points” it is fun to read, Jim is funny, full of wit and humor.

Reply

Roel de Leeuw March 16, 2010 at 5:03 pm

In my database with over 2000 rated whiskies, only 0.2% of the whiskies rate 95 or higher, that is 7 drams out of 2000+.

For what it is worth.
Roel

Reply

Oliver Klimek March 16, 2010 at 5:18 pm

Thank your for this insight, Roel. It is nicely in line with what I expected.

Reply

666ppm May 31, 2010 at 11:25 pm

Very useful, thanks for your thoughts!

I also use the nose/taste/finish/balance system (25/25/25/25) to rate whiskys. And I agree with your view on the compound scoring risk. But for me it is more a mix of “bottom up” and “top down” scoring. “Bottom up” means scoring the four elements nose, taste, finish and balance. “Top down” means calibrating the sum of the four scores with my other whisky ratings and reassessing the four base scores. Why?

Well, when you give a whisky 82 points in your “continuous” system and compare it to an other whisky in your database with 81 points. Why is it 1 point “better”? Is it the nose, the taste, the finish or the balance of it all? I think the compound system can help when using it also a dissection system and identifying the “source” of rating differences.

Reply

Oliver Klimek June 1, 2010 at 1:58 pm

If you are consistent with your ratings and tell the people how you do it, I see no real problems with such an approach. Readers just have to be aware that x out of 100 does not have to mean the same thing for different people.

BTW, did you read my post about the Spirits Spider? Now that’s compound scoring taken to the extreme.

Reply

666ppm June 1, 2010 at 6:05 pm

The Spirits Spider is really funny…..

Reply

Jeff April 1, 2013 at 9:19 am

I respectfully disagree on the answer to “Is this Book any Good Then?” (although I agree Jim Murray’s writing has some merit, so “any”, taken to extreme, makes it a debateable point) and the problem lies in how seriously it takes itself, and aspires to be taken. If you promote yourself as the first and last word on whisky, as Murray does, it just isn’t acceptable on any level that your ratings are inconsistent or demonstrably weighted to a positive result, as is widely acknowledged by other experts, but that Murray’s product should somehow get a pass on flair and bravado. After reading your detailed dissection of the many problems with Jim Murray’s scoring, I find it unexplainable that your conclusion is that “The Whisky Bible is a great collection of tasting notes AND SCORES for thousands of whiskies.” On the page, Jim’s scores all look the same and the only way to sort out the sound from the just plain silly is to rate all the whiskies yourself – hardly a practical stance (or reality) for a book that is supposed to guide people in purchasing. Enjoying a book is not the same as being able to count on it as a reference – and it claims to be the latter.

Reply

DramGoodChemist June 21, 2013 at 5:32 pm

Thanks for the words, Oliver. An excellent read and one that will certainly allow this new convert to Scotch a wider, wiser point of view. In response to the above post: Jeff, if Jim’s Bible is not a go-to reference in your eye, what book(s) instead would you recommend? Is there anyone alive out there claiming to have as much authority on a wide range of whiskies as Mr. Murray in an annually updated format?

Reply

Jeff June 21, 2013 at 10:37 pm

Thanks for the response DramGoodChemist, but to my knowledge, alas, no, there is no such (other) book out there, which is annually updated and goes into such depth – which is the shame of the whole situation, really, that, with his level of knowledge, Mr. Murray keeps pumping out the Whisky Bible with its obvious inconsistencies, when the need for an improved product of this type is so pronounced. And, to come full circle, it’s those inconsistencies which force me to question Mr. Murray’s authority on the subject: I’m no expert, but when JW Red Label is 87.5 and JW Blue Label is only 0.5 above it at 88, and both are eclipsed by Black Grouse at 94, something just isn’t right. I personally believe that Mr. Murray’s ratings, because they are so high overall, might well be the ones which, for obvious reasons, are the most quoted by retailers by a large margin. Furthermore, I’m afraid that being quoted, and the cross marketing that represents, has become the point for Mr. Murray and the Whisky Bible, which is another reason I can’t trust it – it has an agenda other than steering consumers in the right direction.

Michael Jackson’s Malt Whisky Companion certainly deserves mention in any discussion of whisky guides, but the problem is, with Jackson’s passing in 2007, this work will become more out of date with the passing of time – or, with updating, will reflect opinions other than Jackson’s. Serge Valentin, whose profile in on this site, could write an instant best-selling whisky guide, but he doesn’t have to because his more than 8,900 rated whiskies and notes are published for free on http://www.whiskyfun.com/. Without intending any offense to Oliver or any of the other Malt Maniacs, this is, in my estimation, this is the single best place to go for one expert’s opinion. Three other sites I consider pretty dependable go-tos are Malt Maniacs Malt Monitor, the Los Angeles Whisk(e)y Society page, and although it represents contributed reviews from a wide variety of sources, amateur and expert, Whiskybase.

Sláinte! and Happy Dramming.

Reply

Leave a Comment

Previous post:

Next post: