"Stat gurus" believe that "advanced basketball statistics" provide all of the information one needs to completely understand the NBA at both a team and an individual level, while most NBA scouts and personnel directors say "The eye in the sky does not lie"--or, as Indiana Pacers scout Kevin Mackey told me, the "eyeball is number one." I have long criticized "stat gurus" for covering their work with a pseudoscientific patina that obscures the reality that they do not rigorously test their hypotheses or even provide margins of error for their supposedly flawless calculations--but it seems like this season may provide at least an informal test of some of the "stat gurus'" methods and conclusions. It is important to distinguish between baseball's advanced statistics and "advanced basketball statistics"; baseball is a station to station game consisting of discrete actions that can be much more accurately described statistically than a basketball game can, because basketball involves 10 players simultaneously moving and interacting: this is akin to the difference between writing a computer program to play checkers (computers have "solved" checkers already) and writing a computer program to play chess (computers play chess very well but are not even close to "solving" the game), though computers play chess much more proficiently than "stat gurus" are currently able to evaluate basketball productivity at an individual level.
Most of the "stat gurus" have insisted for several years that LeBron James is by far the best player in the NBA and at least some of them would also rank Dwyane Wade as the second best player. Putting James and Wade on the same team instantly caused "stat gurus'" computers to overheat (no pun intended); by their calculations, such a duo would be so "efficient" that it should dominate the NBA even accompanied by three stiffs--but of course the Miami Heat also have Chris Bosh (one of the NBA's top 15 players according to both "stat gurus" and traditional talent evaluators) plus a solid cast of role players. As I said--tongue only partially in cheek--before the season began, if the "stat gurus" are honest they have to admit that their formulas predict that the Heat will win 90 out of 82 regular season games; after all, they have been telling us for years that adding James to the Lakers would have produced the best team in NBA history. I certainly expected--and still expect--for the Heat to be a very good team this season but in my Eastern Conference Preview I correctly identified some of the Heat's important weaknesses and refuted the ludicrous predictions that the Heat would cruise to 70-plus wins en route to easily capturing the NBA championship.
We are nearly one fifth into the 2011 NBA season and the Miami Heat are currently 9-8, sixth in the East and just one and a half games ahead of the Cleveland Cavaliers, a team that the "stat gurus" said would be terrible without James and that would only win 12 games according to one yahoo.
Before Miami's November 24th loss to Orlando, ESPN's "stat guru" Tom Penn declared that the Heat's fans should not be worried despite their team's pedestrian record because the Heat had posted outstanding points per possession numbers both offensively and defensively. Penn asserted that the Heat's exceptional efficiency according to these metrics strongly indicated that the Heat would prove to be an elite team. Jon Barry asked Penn why the Heat have a mediocre record if they are such an efficient team and Penn responded that the Heat's record is merely a reflection of a small sample size of games. Penn added that the Heat had lost a couple of close games and that in the long run he expects LeBron James and Dwyane Wade to carry the Heat to victories most of the time in such games.
Penn's reasoning makes no sense: why should the sample size of (at that time) 14 games be considered small in terms of the Heat's won/loss record but be considered large enough to make a meaningful judgment about the team's so-called efficiency? The reality is that what both the won/loss record and the "advanced basketball statistics" show is that the Heat have blown out inferior teams and have played well at home but they have lost against good teams and they have not played well on the road; in layman's terms, the Heat are frontrunners who have yet to display the necessary mental and physical toughness to be an elite team.
Penn's faulty logic is an excellent example of one of the flaws of "advanced basketball statistics" that I have hammered away at for years: the "stat gurus" rarely if ever provide margins of error or any other kind of guidelines outlining the limitations of the numbers that they cite. The "stat gurus" act like everyone should accept such numbers as 100% accurate and completely incontrovertible. Five years ago, I had a protracted online debate with Bob Chaikin--a "stat guru" who is equally notorious for his sharp tongue and for his narrow, dogmatic views--about Antoine Walker's value. Chaikin, relying on nothing more than "advanced basketball statistics" generated by his basketball statistics simulator, insisted that Walker was the worst starting power forward in the NBA, while I said that just as it is obvious that Walker is not the best starting power forward it is also obvious that he is not the worst. After Walker rejoined the Boston Celtics, he performed quite well and helped lead them to the playoffs. I politely asked Chaikin to explain the dichotomy between what his simulator says and what everyone could see; just like Penn used sample size as an excuse to disregard the Heat's obvious problems, Chaikin dismissed Walker's contributions because of an allegedly small sample size (though Chaikin erroneously referred to "small sample population size"--it is surprisingly common for "stat gurus" to be uninformed about the most basic terminology and concepts regarding the science that they are allegedly practicing). I countered by asking Chaikin to define what an appropriately large sample size would be and challenging him to be willing to admit that he is wrong if Walker's production exceeds Chaikin's expectations once that sample size is filled. Naturally, Chaikin could not produce a valid response to this and so he resorted to childlike name calling, an all-too common reaction in the blogosphere; the jarring thing about my discussion with Chaikin was not his insistence that he was right about everything but rather his refusal to accept even the possibility that he could be wrong. After that exchange, I realized that "stat gurus" are practicing a faith-based religion and not engaging in the scientific inquiry that they allege they are doing.
Elizabeth Kubler-Ross described the stages of grief and I have discovered the stages of dialogue between a "stat guru" and an unbiased, informed basketball analyst:
1) The "stat guru" makes a bold statement/prediction that contradicts what intelligent observers see.
2) When an intelligent observer points out flaws/limitations regarding what the "stat guru" said, the "stat guru" says that the observer is simply allowing himself to be biased by what his eyes see; the "stat guru" insists that the numbers are 100% correct and cannot be refuted.
3) When the intelligent observer then posits some rational, alternative interpretations of the numbers, the "stat guru" resorts to insulting the observer.
This is why I stopped posting at APBR Metrics, the church for the believers in the religion of "advanced basketball statistics"; instead of being a "heretic" inside their small church, I prefer to simply refute their pseudoscience in an open forum.
The intriguing postscript to this story is that Walker eventually landed in Miami, where he started all 23 playoff games in 2006 and ranked second on the team in playoff mpg as the Heat won the championship. Naturally, Chaikin and other "stat gurus" continued to insist that Walker was a terrible player even though Hall of Fame Coach Pat Riley made Walker an integral part of the Heat's rotation.
I regret that I wasted so much time trying to "convert" a rabid cult member like Chaikin but I am glad that--unlike him--I sought the truth, ultimately writing an article (The Enigmatic Antoine Walker) that presented a very balanced account of Walker's strengths and weaknesses as a player. Walker is hardly my favorite player--"stat gurus" disregard the most basic scientific practices but they love to accuse their critics of being biased--and I certainly have never in my life been a Celtics fan; what intrigues me about Walker is the seemingly visceral hate that "stat gurus" direct toward him. "Stat gurus" seem to be personally offended that any NBA coach would put Walker into a game and that is why I used Walker's return to Boston as an opportunity to try to understand why "stat gurus" hate Walker so much and to initiate a conversation about the strengths and limitations of "advanced basketball statistics"; it seems obvious that if the "stat gurus" say that Walker is the worst starting power forward in the NBA but he ends up playing a key role on a championship team than the "advanced basketball statistics" are not telling a completely accurate story. Dan Rosenbaum is one of the few "stat gurus" who appreciated the nuances of my comments; he realized that I was not trying to pump up one player but rather to critically examine "advanced basketball statistics" as a whole and he understood that "stat gurus" must effectively answer the kinds of questions I raised instead of shrieking ad hominem insults a la Chaikin.
The Walker episode shows us that "stat gurus" have been wrong before and that they refuse to admit that they are wrong even in the face of clear empirical evidence. So, don't expect Miami to win the championship this year--and after Miami is eliminated from the playoffs don't expect the "stat gurus" to admit that they were wrong about James, Wade or anything else. After all, a "true believer" never lets reality get in the way of his "faith."
-----
Postscript:
The disturbing trend is not that a few "stat gurus" talk amongst themselves at APBR Metrics but rather that they have brainwashed some mainstream media writers (Henry Abbott) and outlets (Wall Street Journal) into uncritically accepting their declarations as gospel. I am certainly not an anti-statistics Luddite; I have always loved numbers and I look forward to the day that someone comes up with a way to apply statistics as meaningfully to basketball as they have been applied to baseball but it is very important to not blindly accept anything uncritically. Albert Einstein's Theory of Relativity may be the most successful theory in history, yet physicists are still constantly testing it to make sure that it is the best possible description of reality. If "stat gurus" were truly scientists and not just cult preachers then they would welcome intelligent questions about their ideas, but "stat gurus" not only possess little evident capacity for self reflection/self criticism regarding their "advanced statistics" but they appear to have no interest in research showing that assist numbers may be inflated--and it should be obvious that no matter how good a formula is its conclusions will be meaningless if the basic data (culled from box scores) is inaccurate.
After I did my research about Chris Paul's inflated assist totals, David Biderman of the Wall Street Journal contacted me to learn more about this subject--but he did not quote me in his subsequent article and instead cited David Berri, who has since apparently become one of Biderman's chief sources (Biderman can scarcely write two words about the NBA without mentioning Berri). I asked Biderman why he did not even mention my article about Paul's assists (Biderman had initially told me that my article inspired him to write his own article) and Biderman replied that he did not have enough space in his article to do justice to my in depth analysis. In other words, Biderman and/or the Wall Street Journal are not willing to allocate enough space to provide a proper, complete discussion of the flaws regarding current NBA scorekeeping and the limitations of basketball statistical analysis but they are perfectly willing to provide a forum for Berri to repeatedly issue pithy, misleading and inaccurate sound bites!
That tells you all you need to know about "advanced basketball statistics" and about how the mainstream media functions.
Penn is not Mightier Than Eye in the Sky
