xG LIMITATIONS
(March 2018)
My projection method is built on goals prevented over expected (I say goals prevented, rather than saved because I think it’s important to remove credit for saves made on preventable rebounds), which measures the difference in actual goals allowed by a goalie and the number an average goalie would. In theory, if we can control for number and quality of shots faced we should be able to isolate and identify puck stopping ‘skill,’ the marginal difference between the best and the rest would manifest itself over an adequate number of shots.
This means it falls on the analyst to properly adjust for shot quantity and quality. I mention quantity because in theory goaltenders can discourage shots against and encourage shooters missing the net through excellent positioning. However, I’m not fully confident there’s a convincing way to separate the effect of a goalie forcing misses and playing on a team where the scorer might be more or less likely to record a missed shot. These effects don’t necessarily persist season-to-season, so I’m still using all non-blocked attempts in my analyses, but it’s important to acknowledge what should be a simple counting statistic is much more complex beneath the surface.
A more popular contention is shot quality, where the analyst tries to weight each shot by its probability of being a goal by holding constant the circumstances out of goalies control that can be quantified using publicly available NHL play-by-play data – things like shot location, team strength, shot type, prior events (turnover, faceoff, etc) and their locations. But these factors, as skeptics will point out, currently doesn’t include shot placement, velocity, net-front traffic, or pre-shot puck movement involving passes
All of this is to say the ability to create a ‘true,’ comprehensive, and accurate xG model is limited by the nature of the public data available to analysts.
On the surface, the expected goals assigned to each of these goals are low. A lot of this is confirmation bias – we see the goal, so the probability of a goal must have been higher, but in reality, pucks bounce, goalies make great saves, and shots miss the net. However, we’ve identified additional factors, that if properly quantified, would increase the probability of a goal for those specific shots.
All teams gameplan to generate traffic and cross-ice passing plays, but some have the personnel and talent to execute better than others, while some teams have the ability to counter those strategies better. Some teams will over or underperform their expected goals partially due to these latent variables.
Unfortunately, there isn’t necessarily the data available to quantify how much of that effect is repeatable at a team or player level.
So while a well-designed xG model can implicitly capture some effect of pre-shot movement and other latent factors, it’s safe to say team-level expected goal biases exist.
Final Thoughts
Model testing suggests that even a good projection of goaltender performance would have a healthy margin of error. Misses will happen, but they should be as small as possible and be accompanied by a margin of error.
LIMITATIONS OF IMPACT MODELS
What Are Legitimate Criticisms of “Impact” Models?
Some of the most well known impact models include Evolving-Hockey’s RAPM, TopDownHockey’s RAPM, and Hockeyviz’s Impact Model “Magnus”. These models all determine the value of a players shifts in terms of what happened in terms of goals/shots versus what we would’ve expected given the score, venue, rest, zone, teammates, and competition.
But, there are things that the model can miss. And it helps to understand what they do, so we can understand what they don’t do.
“Impact” models quantify player impact implicitly, not explicitly
All of the modes use “on-ice” metrics exclusively. You may think of a players “impact” as the value — in terms of goals, shots, etc. — of what they do on the ice. But it’s important to recognize that these models do not measure what a player does, they only measure what happened. So they moreso represent the value of a player’s “shifts” than the value of a player’s “performance.” While counterintuitive, this is actually by design. We don’t care what a player does unless that thing will help produce an outcome we care about. So, rather than measuring the things they do, we measure the outcome. How they produce that outcome is not always clear.
The models attribute “credit” based on what happens in shifts. Their guess about who deserves credit will improve over time as it learns.
All impact models are different
To elaborate:
· The models work on the assumption that all players react to circumstances in roughly the same way.
· Another thing these models don’t do is use the individual performance of a player to inform their “guess” about who deserves credit.
· The models also don’t know that the shift ended in a more advantageous position than it began. It measures “success” only in terms of the offensive and defensive count of a given metric (typically shots, xGs, or goals).
· Lastly, the models don’t know a lot of specifics about the players.
댓글