Plus/minus seems like a perfectly objective statistic and a very useful one for people who cannot watch every single minute of every single game (which would describe everyone, even though one blogger likes to pretend/brag that he alone has consumed every second of NBA action since sometime during the Bulls' three-peat glory days): you simply calculate the scoring differential when each player is on the court and, voila, you can see which players are having a positive impact and which players are having a negative impact; all of the hustle plays that are not captured by conventional box score numbers or even by "advanced stats" are presumably detected by plus/minus. Furthermore, adjusted plus/minus--which takes into account who else is on the court--seemingly refines the data even more.
However, anyone who has looked at a lot of plus/minus data immediately realizes that it is very "noisy": some players who even the "stat gurus" know are not that good inexplicably have impressive plus/minus numbers, while some players who are clearly above average do not have outstanding plus/minus numbers. At the very least it seems obvious that one needs a very large set of data to filter out this noise. "Stat guru" Ken Pomeroy recently devised a very interesting test of the limitations of plus/minus; his work focused on the college game but can clearly be applied to the NBA game as well. You can read a detailed description of his methodologies and results in A treatise on plus/minus but his conclusion should be embraced by anyone who is attempting to analyze basketball: "It's true plus-minus captures everything that's happening, but that includes a whole lot of random things that lead to a hoop or a stop. Things that have nothing to do with the ability of the player you want to analyze. In basketball analysis, we should be filtering out randomness, not embracing it." Pomeroy notes that because the professional season is much longer than the college season there may be "limited use" for adjusted plus/minus in the NBA but even in that case one probably needs at least two full seasons of data to make any meaningful evaluations; in other words, most of the stat-based articles (about "clutch performance," player ratings, MVP rankings, etc.) that are popping up like dandelions in an untended yard are using data sets that are far too small to form the basis for sweeping, definitive conclusions (I realize that not all of these articles are using plus/minus or advanced plus/minus data but there is even less reason to trust the accuracy of Berri's numbers or Hollinger's numbers--both of which are based on subjective formulas that can be tweaked to reach whatever conclusions the author desires--then there is to trust plus/minus data that truly is objective in some sense even if it is only potentially meaningful when the data set is very large).
Pomeroy's article represents the kind of frankly honest research/experimentation that all "stat gurus" should be doing; instead of brazenly declaring that their numbers are flawless while the observations of skilled talent evaluators are hopelessly biased, "stat gurus" should be in their labs (metaphorically speaking) trying to ascertain the strengths and limitations of their beloved formulas: if more of them would do that--instead of writing articles with catchy headlines so that certain high profile entertainment providers will link to them--then they could make a real contribution to better understanding basketball.
Ken Pomeroy Describes the Limitations of Plus/Minus
