As I'm sure most of you have noticed, we often refer to the tempo-neutral statistics you can find at Ken Pomeroy's site, kenpom.com, such as efficiency, rebounding percentage, turnover percentage, etc. If you don't understand why we use those stats or what they are, you can read my best stab at a plain language explanation here.
We also often will refer to his rating system, which seems to have become a point of consternation and confusion for some people, so I'm going to try and bring a little clarity to the discussion.
First off, before we even talk about whether the ratings have any merit or not, the first thing you have to understand is what his rating system is trying to do. Here's his premise:
The first thing you should know about this system is that it is designed to be purely predictive. If you’re looking for a system that rates teams on how "good" their season has been, you’ve come to the wrong place. There are enough systems out there that rank teams based on what is "good" by just about any definition you can think of. So I’d encourage you to google college basketball ratings or even try the opinion polls for something that is more your style.
The purpose of this system is to show how strong a team would be if it played tonight, independent of injuries or emotional factors. Since nobody can see every team play all (or even most) of their games, this system is designed to give you a snapshot of a team’s current level of play.
That's sooooooo very important to keep in mind when we discuss these ratings. It's a great piece of information when talking about the relative strength of teams, or how teams have played so far. When we're trying to figure out whether the Cougs are really that much better of a team than their opponent, or whether a 28-point victory over Mississippi Valley State really means much -- the answer is no, because most everyone beats the Delta Devils by a lot of points -- it's solid.
But it's not a great piece of information when discussing certain other things, such as who's got the better tournament resume, or whatever. That's because it's not meant to be. Since a team's season is determined a success by whether a team advances to the NCAA Tournament and how far it advances in that tournament, results clearly do matter. But this system doesn't reward wins and losses per se:
I would describe the philosophy of the system as this: it looks at who a team has beaten and how they have beaten them. Same thing on the losses, also. Yes, it values a 20 point win more than a 5 point win. It likes a team that loses a lot of close games against strong opposition more than one that wins a lot of close games against weak opposition.
This seems to be the sticking point for a lot of people -- after all, isn't whether you win or lose the most important thing? -- but that's precisely why we like to use it: In the small sample size world of college basketball, where the talent level of opponents varies so wildly, wins and losses often aren't the best measure of how strong a team is (as the RPI so very clearly shows). And that's what we're usually trying to figure out when we do refer to Pomeroy's ratings, so it fits.
Here's a good example: 2008 West Virginia. The Mountaineers were 24-10 heading into the NCAA Tournament and given a No. 7 seed after finishing sixth in the Big East with an 11-7 record. However, in addition to their wins, they had a number of close losses to some good teams, so Pomeroy had them rated No. 17 overall. All the Mountaineers did was win their first game, then knock off No. 2 seed Duke in the second round before losing in overtime to No. 3 seed Xavier in the Sweet 16. They were portrayed nationally as a Cinderella story; those of us that follow Pomeroy's ratings weren't really surprised.
Same thing with the Cougs that year. Many were surprised at the dominating fashion with which they disposed of their opponents in the first round, but again, the Cougs were rated No. 10 by Pomeroy, while Notre Dame was rated 28th, despite them only being separated by one seed line in the tournament. That the Cougs handled the Irish so easily shouldn't have been a shock.
But as the next game against North Carolina proved, there also has to be room for using your brain. Pomeroy predicted a close loss to the Tar Heels, as they were ranked No. 4 to our No. 10 in his system. But we all knew there was the potential for a blowout, given the Heels' advantage in athleticism which would be magnified by something as simple as a cold shooting night by WSU. That's exactly what happened, and obviously something number crunching can't predict.
So is it perfect? Heck no -- no statistical system is, especially not one based on such small sample sizes (right now, anywhere between eight and 12 games). But it's a useful piece of information when placed in its proper context.