clock menu more-arrow no yes mobile

Filed under:

On Belichick's decision (or, Why results-based analysis is stupid and how it applies to WSU)

If you buy something from an SB Nation link, Vox Media may earn a commission. See our ethics statement.

I realize I'm a little late to this party, but I started this yesterday and finally was able to get it finished. Besides, the point isn't necessarily Belichick -- it's what the decision represents. So without further ado ...

Unless you live under a rock, you've heard that the Patriots unsuccessfully went for it on 4th-and-2 from their own 28 while up by six points with just 2:08 remaining on Sunday Night Football against the Colts. And -- again -- unless you live under a rock, you've no doubt heard the lambasting New England coach Bill Belicheck has received at the hands of the ubiquitous "media" for his now dubious decision.

Except that I don't think it's all that dubious.

I'll readily admit that as I watched the Patriots line up, I openly questioned it. And after the play failed, I questioned it further. But then, in all of my open questioning, I said this: "I mean, I can understand how he reached that decision, but I just don't get it." That made me think.

Was I questioning the decision, or the result?

At first blush, the difference might seem insignificant. After all, as the mainstream media likes to remind us all -- every second of every day -- sports is a bottom-line business. Either you win or you don't, and the reasons you win or don't win are really only useful for talk show fodder.



But questioning the decision (which is based on a process) and the result (which is usually based on an incalculable number of variables completely out of the control of the decision maker) are two different things entirely -- much different than most people ever want to recognize. After all, if the decision doesn't generate the desired result, someone has to be to blame, right?

I'm reminded of this matrix from "Winning Decisions" by Russo and Schoemaker, introduced to me by Paul DePodesta:

GOOD RESULT BAD RESULT
GOOD PROCESS Deserved Success Tough Break
BAD PROCESS Dumb Luck Poetic Justice

I would argue that Belichick's decision is squarely in the upper right corner. Here's why.

A number of you -- like the media -- would argue that Belichick's decision was pure folly, actually decreasing the Patriots' chances of winning the game. That's simply untrue. Brian Burke explains why at The Fifth Down, but I like Joe Posnanski's simplified summary of Burke's writing better:

His explanation is simply this: A team picks up fourth and two about 60 percent of the time — and we all know that a fourth down conversion in this case means certain victory. On the flip side: A team would score a game-winning touchdown from the 30 about 53 percent of the time. This leads to this formula — the first part is the 60% multiplied by 1 (1 signifying the certain victory if the play is converted). The second part is 40 percent multiplied by the chance of winning the game if the 4th down play fails:

(.60 *1) + (.40*(1-.53)) = 78.8% chance of winning.

There you go. Burke then estimates the chance of winning if Belichick punts — that is the chance of a team going 66 yards for a touchdown in the final two minutes. He says, historically, teams get that about 30% of the time. So a punt gives the Patriots a 70% chance of winning.

And there you go — 78.8% chance of winning vs. a 70% chance if you punt. It really is clear cut. I don’t know if Belichick plays with such percentages in his mind, but instinctively he knew that his team’s best chance to win was to go for it. 

Posnanski anticipates that some would argue that the Colts' chances of scoring from the 30 are higher than 53 percent, but as he rightly points out, they also have a higher chance of scoring from 66 yards, and the Patriots also have a higher chance of converting that 4th down than the average team. Essentially, it doesn't matter how you try to rig the numbers, the statistical conclusion has to remain the same.

But, of course, if football were dictated by odds and probabilities, teams would go for it on fourth down every time. But that's not the sport we watch. We watch a sport where coaches make millions of dollars a year, and any unconventional decision puts that livelihood at risk. So, conventional wisdom reigns.

Thirty one coaches would have punted on 4th-and-2 from their own 28 with 2:08 remaining.

One didn't.

Which brings me back to that matrix. Some might argue that the decision to go for it was bad process. I'd say you're not thinking big enough.

Let me ask you a question: If the Patriots convert there, are people roundly roasting Belichick for the following 48 hours? Unlikely. People might scratch their heads, but more than likely he's being praised for a gutsy move that put a game away. And that's the flaw in results-based analysis. Either the decision was the right one, or it wasn't; it shouldn't matter what the outcome was, especially when the preponderance of evidence -- years and years of good results -- suggests that the process is a good one.

Since the institution of the salary cap, no franchise in the NFL has been more successful than the New England Patriots. I'd argue they've had a pretty good process that has allowed them to be so successful. It's a process that's predicated on playing like a champion every minute of every day, whether in practice or in games. Champions are ready to take a game at any moment, no matter the game situation. Champions have an unwavering belief in their ability to do what they do.

Bill Belichick's process -- which undeniably involves creating a culture of belief in the "Patriot Way" -- has resulted in three Super Bowl victories and a 16-0 regular season. That would seem to qualify it as a good process. Did it all of a sudden become a bad process because of the result of one play? A play that failed only because a guy who normally has one of the best sets of hands on the team bobbles a catch? You can question whether it was prudent to go for it there, but to so quickly ridicule it as indefensible shows just how much safety people feel in their conventional wisdom.

If the process is good, you're more often than not going to end up in that upper left corner -- conventional wisdom be darned. But sometimes, through no fault of your own, you end up in the upper right. That doesn't make the process bad. Especially if you have years of good results to back up the process.

Which brings me around to the Cougs.

When Dick Bennett came to town, he brought a mantra: Trust the process. Let me build the team my way. Have patience. The process always works. And he had years and years of good results to back it up. When his boys lost by about 100 points to Oklahoma State, outside observers questioned the progress -- but Bennett and the players did not. Little did we know that good results were just right around the corner.

So, this is the dilemma Jim Sterk finds himself in with regards to the football team: How much does he trust Paul Wulff's process? Sure, it worked at Eastern Washington; will it work at WSU?

If we're to look at Wulff's process, we'd have to say it's got six steps:

  1. Weed out the bad apples, even if it means exacerbating poor results;
  2. Get those left to buy into the program through training and nutrition;
  3. Bring in your own quality guys;
  4. Coach all of them to trust their systems, even as they experience poor results;
  5. Repeat Steps 2-4 for two or three years;
  6. Expect good results that are now sustainable across multiple years.

This, of course, leaves Sterk with two questions. First, is this process even guaranteed to get you to step six at some point? And if you think it is, can you wait long enough to get there without completely alienating the fan base?

Through this lens, it becomes pretty clear what Wulff's biggest problem is. It's not that the team is getting killed every weekend. That stinks, but it's not a coach killer. No, his biggest issue is that fewer and fewer people are trusting his process with each passing week. He doesn't have a track record to point to like Bennett did. A statement like, "This is how we did it at Eastern and we believe in it," simply isn't a convincing argument for his process. Fair or not, nobody cares what you did at an FCS school. This is the land of the big boys.

Sterk's got some big decisions to make this offseason. Does he believe in the process? If he doesn't, he's left with no choice but to fire Wulff. However, statements from Sterk himself suggest that he does believe in the process, overall. Now, if he believes the process is good overall, but has some questionable aspects, then he has to step in and demand change. Because when the results are as bad as they have been -- as consistently as they have been -- there's really no way you can reach the conclusion that the process is just fine.

Somehow, some way, we've got to get ourselves back in that upper left-hand corner -- or, at the very least, convince fans we're living in that bottom right corner.