Methodology
UmpScores was founded on the simple principle of improving the game of MLB baseball by providing greater transparency and measurements of home plate umpire performance.
Bad Call Ratio
Umpire Accuracy
The Bad Call Ratio (BCR) is one of our main metric for evaluating umpire accuracy behind the plate. Simply put, it is the number of incorrect calls (either a ball that should have been called a strike, or a strike that should have been called a ball) divided by the total number of calls made by the umpire. And pitches in which the batter swung are not used.
To accurately measure our ratios, UmpScores has custom-built player strike zones based on data available on Statcast. We then chart each pitch in an at-bat against the player specific zone to determine if the pitch is inside or outside of the zone. On close calls, we run simulations to see how a pitch might be called in similar situations. This allows us to remove any uncertainty created by the Hawkeye tracking system. This system ensures that we give the benefit of the doubt to umpires on tough calls and guarantees an umpire will not be credited with a wrong call simply because the pitch was measured incorrectly.
Every umpire has their own BCR. Our umpire rating system, on a scale of 1 to 5 stars, ranks umpires based on their ball/strike calling performance relative to their peers. 5-star ratings are reserved only for the very top umpires, it is the gold standard of performance. When such an umpire is behind the plate for that game, you know you are more likely to see fewer missed calls that game.
Umpire Precision & Consistency
Measured By Our Consistency Metric
While the BCR covers umpire accuracy, umpire precision, most commonly referred to as consistency, is measured differently.
UmpScores attempts to correct a flawed approach. Most measures of consistency typically create umpire game zones based on the outer bounds of called strikes called by the umpire, whether correct or not. Any called ball within this zone is then marked as inconsistent. We view this often used approach as flawed as it measures consistency based on outliers. These calls are the exception, not the norm.
To correct for this observed flaw, UmpScores uses machine learning techniques to create umpire zones that more effectively capture consistency by removing outliers which are not representative of an umpire’s zone.
Critical Calls
Performance In Critical Situations
Some errors behind Homeplate are more detrimental than others. An incorrect ball when the count is 1-1 might not matter if the batter pops up on the next pitch. But when such a call is made on a 3-1 count, that batter automatically earns a walk. Worse, if the bases are loaded, a run comes in. This has a much greater effect on the game and deserves critical attention.
UmpScores has prioritized each situation in a game based on pitch count, runners on base, and number of outs to determine which calls an umpire made- good or bad- have the greatest influence on the game. When we elect to highlight a missed critical call by an umpire, it is because that situation was pivotal to the game. We also highlight strong performances- when umpires get all their calls right in critical situations. In such circumstances, these critical calls will be denoted and highlighted on the UmpScores scorecard.