"Numbers Never Lie. " That’s what ESPN tries to sell viewers daily at 4 p.m. Eastern on ESPN2, with a recently-revamped show by that title that features Michael Smith, Jalen Rose and Hugh Douglas supposedly talking statistics. They come up with some rather peculiar ways of doing so, including this video posted to YouTube Friday, which pronounces San Francisco 49ers quarterback Alex Smith to be the best quarterback in the NFL and (in the YouTube description) says he "belongs on the NFL's Mount Rushmore of Quarterbacks." Watch it for yourself:
This is not comedy, although Alex Smith's 200-passing-yard, zero-touchdown, three-interception performance in a 26-3 loss to New York Sunday may make it seem that way in retrospect. This is a serious analyst on a serious show that bills itself as being about the numbers anointing Alex Smith as the NFL's top quarterback. In fact, the Mount Rushmore line in the video description (and presume it isn't a humorous one like our own Mount Rushmore), would seem to imply Smith is one of the best four quarterbacks at the moment.
How do "the numbers" lead Michael Smith to that conclusion, one that's patently ludicrous to most NFL observers? Heck, even 49ers fans don't buy it, and many didn't buy it before his meltdown Sunday. Keep in mind that the 49ers flirted with Peyton Manning this summer, and some fans were frustrated when they didn't get him. Moreover, heading into this game, sites like Accuscore said "The bread and butter of this San Francisco offense is the run game," and projected him for just 217 passing yards. Yahoo fantasy analyst Scott Pianowski questioned just how well Smith can run a passing-focused offence before Sunday's game, and debate partner Michael Salfino's rebuttal was the less-than-thorough praise of calling Smith "at least above average." Thus, Michael Smith isn't drawing this from a groundswell of popular opinion.
What about "the numbers"? Well, Alex Smith also was nowhere near the top of most statistical categories widely considered for quarterbacks, including passing yards, completion percentage and touchdowns, and even his 8-1 touchdown/interception ratio wasn't the league's best heading into this week (that would be the 9-1 mark Pittsburgh's Ben Roethlisberger had before throwing his second interception of the season Thursday). He did lead in two categories, though; quarterback rating (also known as passer rating), which is prominent but has been widely critized, and ESPN's own proprietary Total QBR, which has been bashed by an even wider array of commentators, analysts and players (including Aaron Rodgers), and seems to largely be used by people who work for the Worldwide Leader. Unsurprisingly, that's the number Michael Smith chose as the centerpiece of his case for Alex Smith.
The problem isn't that the Total QBR number itself is intentionally out to decieve. The famed comment "There are three kinds of lies: lies, damned lies, and statistics" (which is oft-attributed to Benjamin Disraeli, but has uncertain origins of its own) is just as much a fallacy in its own way as "Numbers Never Lie", but it does make a good point. Yes, numbers can't talk and can't directly sell you a bridge themselves, but any data point can be selectively used to lead towards almost any conclusion. As New York Times polling guru and long-time baseball stathead Nate Silver explores in his new book, The Signal and the Noise, the challenge is trying to figure out how the numbers fit together. Which ones are signal (useful data), and which are just misleading or distracting noise?
It's not that conclusions that go against the popular perception are always wrong. Heck, that's what led to the success of the Oakland Athletics' statistics-driven approach described in Moneyball; focusing on low-average, high-on-base-percentage guys in particular allowed Billy Beane to exploit a market inefficiency. The problem is when you start selectively focusing only on data that supports your case and ignoring data that undermines it. For example, even if we ignore Total QBR's problems for a moment, it's still just one statistic; Smith's deficiencies in yards, touchdowns and other areas still should be considered before you anoint him as the best quarterback in the league, and even if his Total QBR is seen as good enough to ignore those other stats, his strength-of-schedule to date is also worth looking at.
Before Sunday, Smith had faced Green Bay, Detroit, Minnesota, the New York Jets and Buffalo. Those teams' passing defences currently rank 18th, 15th, 10th, 5th and 20th in the NFL in passing yards allowed per game. Sure, that's not a complete lineup of bottom-feeders, and sure, Smith's performance against them factored into where they currently stand, but that context doesn't completely endorse him as the league's best, especially when you consider that he'd never been seen as even close to that status before this year (and that his career stats are mediocre at best). Michael Smith cites a couple of other numbers in the video, including Smith's league-high 60.9 percent success rate on passes that travel over 20 yards in the air, but these are relatively minor statistics. Also, if Smith is so great at throwing deep, why doesn't he have more passing yards? (A possible answer is that he doesn't throw deep all that much, so it can take defences by surprise, and that would explain why he hasn't piled up massive yardage totals.) Add everything up, and you get a lot of factors weighing against Smith, with only Total QBR and QB rating really in his corner. As Silver wrote Monday in a post about elections and polling, it's dangerous from a statistical perspective to draw broad conclusions based on a few numbers you're choosing:
Translate that from politics to sports, and you get what happened here. Plenty of statistics weighed against Smith being the NFL's best quarterback before this week, but ESPN's proprietary one said he topped the charts. Sure, as a statistic that combines other statistics, it's not a single data point, but it's still one indicator and should be treated as such (especially given how many top football minds aren't impressed with it), not the be-all and end-all of determining a player's value. Considering it that way is problematic statistically, especially when it's going against what every other indicator says, and it would be a very curious move for a show that's really about statistics.
However, as Smith admitted on a statistics panel at Blogs With Balls in Toronto earlier this month, that’s not entirely the focus of Numbers Never Lie. Here's what he had to say there:
“Numbers Never Lie was initially conceived and dubbed as the fantasy show,” Smith said. “It now is a debate show, like most other shows on ESPN. … I hate to say it’s not about analytics, but it’s not about analytics.” Smith admitted the title’s a little deceiving given the show’s current format. “The show Numbers Never Lie does give kind of a false impression,” he said. “Numbers are the flavour of the show, but not the focus of the show.”
Exactly, and that's how you get cases like this. Saying Alex Smith is the best quarterback in the NFL probably isn't a good move from a statistical show's perspective. It is likely a good move from a debate show's perspective; it's an extreme, unconventional position, and one that's probably going to get a few more people talking than if they'd just discussed typical top quarterbacks like Aaron Rodgers, Tom Brady or Drew Brees. ESPN's exploited that time after time with their Tim Tebow coverage, and funnily enough, Numbers Never Lie is also known for such hard-hitting statistical videos as "Tim Tebow: All He Does Is Win" (about how the Jets this year are totally the Broncos from last year and includes Smith saying "Maybe the Jets shouldn't waste another minute starting the clock on Tebow Time") and "Tim Tebow Will Make The Jets Better" (cherry-picking Tebow's red-zone touchdown percentage as evidence he's a great quarterback, while ignoring all of his many statistical flaws). Numbers Never Lie has also prominently featured ESPN's Rob Parker, and Parker has perhaps trolled harder against advanced statistics than anyone else in recent memory. For a debate show, that's about what many of us expect from ESPN. For something supposedly about stats, viewers should demand better.
The best part of this video coming from Michael Smith? At that same Blogs With Balls panel, on statistics, he said he was personally very much into the statistics movement, but found it hard to sell to the masses. “We’re in the information age with so many people who don’t want to embrace information,” Smith said. “I’m about analytics. … If you’re not embracing analytics, you’re a fool.”" Fellow panelists there also made valuable points about the problems of cherry-picking stats, with Joe Fortenbaugh of The National Football Post saying “Anyone can build an argument and find a stat to support it. The key is to find the right stat.” Funnily enough, another ESPN type on the panel (and one with much greater statistics credibility), Tom Haberstroh, pointed out the central issue here: many solid statistics tend to deal more in probabilities and margins of error than bold predictions, but “There’s the rub, because what sells on TV is bold proclamations.” Those are apparently selling just fine for Numbers Never Lie, even when they're exceptionally dubious statistically. The numbers themselves may not be swindling people, but viewers should ask themselves what cause these numbers are being presented in aid of, and which non-lying numbers are being left out.