An ESPN graphic on their analytics for Will Levis at pick No. 25. An ESPN graphic on their analytics for Will Levis at pick No. 25. (Awful Announcing on Twitter.)
ESPNNFLBy Andrew Bucholtz on

There have been wide debates about the usage of analytics in sports over the last decades, both by teams and on broadcasts. That’s been true at times for even “analytics” numbers that are simple to explain (see the early debates about on-base percentage and OPS, which thankfully have died down a little these days). But it’s particularly true for more complicated analytics, and especially true for those where the model isn’t fully empirical or fully transparent.

The latest case of this came on ESPN’s NFL Draft first-round coverage Thursday. There, they kept showing how their analytics model projected extremely low chances that Kentucky Wildcats’ quarterback Will Levis would still be available at certain picks. And they did so over images of a sad Levis sitting there unpicked:

Levis was not selected at all in the first round in the end, and will be heading home rather than remaining on camera for Day 2. So the 0.1 percent chance (and beyond) came to be. And while a 0.1 percent chance is still a legitimate (if unexpected) outcome, and one that doesn’t necessarily fully refute a model, there are questions about the calculation of that probability and ESPN’s reliance on repeatedly broadcasting it. Here’s some of the criticism that took:

Of course, ESPN’s analytics are far from the only public comments that thought Levis would be selected high. Many prominent media members’ mock drafts had him going as high as pick No. 4 (to the Indianapolis Colts, who did choose a quarterback, but picked Florida Gators’ QB Anthony Richardson). And there was even a Reddit post this week that influenced betting markets, claiming that Levis was telling family and friends he’d go at No. 1 to the Carolina Panthers. (Levis himself responded with “Don’t believe everything you read on the internet.”) So ESPN was not the only one hyping up Levis more than the drafting NFL teams ultimately decided he was worth. But as a draft broadcaster, and one using these particular “analytics” as a way to put numbers on these odds, their approach took quite a bit of scrutiny.

Part of the issue here is that these specific “analytics” are not actually based on empirical numbers. ESPN has explained their Draft Day Predictor (which the public can play around with here) somewhat, saying it “uses expert mock drafts, Scouts Inc. grades and team needs to forecast pick probabilities for prospects in the upcoming draft.” But “team needs” are obviously somewhat subjective (yes, a team with a top quarterback is less likely to draft a quarterback, but that hasn’t always stopped them; “team needs” are probabilities and estimates, not observable numbers). And so are those expert mock drafts (and it’s not clear which ones are included) and Scouts Inc. grades.

It’s of course possible to turn qualitative analysis into somewhat quantitative analysis by assigning numerical probabilities to outcomes, and there can be value to those models. And this model was not entirely wrong even on Levis. While this predictor was high on him going early, it also thought “the most likely answer is Levis” for a potential slide and said his model had a “long right tail” (so higher chances of going later than fellow QBs Richardson, C.J. Stroud (taken at No. 2 by the Houston Texans), and Bryce Young (taken at No. 1 by the Panthers)).

The ESPN Draft Day Predictor had some other hits, too. That including projecting a 40 percent chance that Alabama RB Jahmyr Gibbs would go in the first round. Gibbs’ No. 12 selection by the Detroit Lions stunned many, but the ESPN model was higher on him than many mock drafts were. And the model may still be correct about Levis being the fourth quarterback taken. The issue was more its thinking that four teams would take quarterbacks in the first round.

The latest

But, as noted, that depends on “team needs.” And that includes teams’ thinking on both how much they want Levis and how much they think other teams might want him. From the outside, without actual knowledge of teams’ plans, that’s a probability estimation, not an observable number. So what showing this “analytics” number on the screen repeatedly was actually saying was “our combination of certain mock drafts, Scouts Inc. grades, and our evaluation of team needs expected Levis to not be available here.” That’s no different than having a draft analyst like Mel Kiper Jr. talk about their surprise a player is still available (and Kiper, who had Levis to the Colts in his final mock draft, did just that as well), and there’s maybe more merit to it as a collection of different opinions and projections rather than one individual’s projection, but it should be seen as that collection of projections rather than actual hard-and-fast data.

Another issue with the repeated references to these particular “analytics” is that they are not calculable from the outside. There’s an edge for advanced stats that fully indicate how they’re calculated, even if they’re complicated calculations (such as Baseball Reference’s version of wins above replacement, or WAR). It’s fully known what goes into that, and, as such, it’s possible for even those who don’t work at that site to figure out a player’s bWAR (and to understand which particular factors are responsible for one player’s being higher than another’s). Similarly, the various “win probability” or “win expectancy” models usually shown on broadcasts or in media coverage can generally be calculated from the outside. By contrast, we know the categories of data that ESPN is putting into this Draft Day Predictor, but we don’t know the specific inputs (which mock drafts are chosen? How are “team needs” calculated?), so no one outside of ESPN can know for sure why the model is putting out the results it is.

Of course, this specific result should be seen in the wider context of ESPN Stats and Info’s analytics work. As mentioned above, this model itself had some successes Thursday, and was even right on Levis being the most likely of the top four quarterbacks to slide. It just potentially underestimated how far that slide could go. (And we say “potentially” because that 1 in a 1,000 chance of Levis being available at pick 25 may have been perfectly valid; a draft is a one-off situation rather than a repeated outcome that averages out, and improbable things do happen, sometimes even one-in-a-million shots.) And that ESPN department has done a lot of impressive work over the years, and has employed a lot of impressive people.

But we have seen ESPN’s broadcasts sometimes overuse their own stats, and use them when they produce improbable conclusions that other stats dispute. And we’ve seen them also sometimes use their own stats with too much ascribed authority, and too little explanation of what they’re actually saying. And the repeated graphics here with Levis (and graphics carry much more long-term weight than just on-broadcast comments) felt like an example of that. And they wound up raising more questions about the ESPN model than about teams passing on Levis.

About Andrew Bucholtz

Andrew Bucholtz has been covering sports media for Awful Announcing since 2012. He is also a staff writer for The Comeback. His previous work includes time at Yahoo! Sports Canada and Black Press.

View all posts by Andrew Bucholtz