Home Boys' Lists Girls' Lists Men's Teams Women's Teams News Photos Contributors Links Help Sign-UpOnline Store
Latest News | Categories | Authors | News Archives

News & Features

Ranking Feature
What Is An Upset?

Since 2015, Tennis Recruiting has analyzed the performance of tennis ranking and rating algorithms for 17 different tennis tournaments - ranging from USTA junior sectional championships to professional Grand Slam events. These tournaments include over 20,000 match results, and a summary of our findings can be found here. Our most recent analysis came from the 2016 Easter Bowl in April. These studies investigate how well the various tennis rating systems predict winners and upsets.

When we release one of these analysis articles, the most common question we field is, "What do you mean by an upset?" There is some confusion as to which matches are labeled upsets in these studies - and also what constitutes a real or true upset.

This article tries to address the question of "What is an upset?" with help from the data we have collected.


Simple Definition

One simple definition of an upset is any match where a lower-rated player beats a higher-rated player. This is a meaningful position, and our analysis articles make use of this definition because it provides a common framework for evaluating the same results across different ranking and rating systems.

While this definition looks easy to understand, it becomes more complicated when we realize that there are different rating systems that rank players differently. A player who is highly-rated by one system might be rated lower in another.

When addressing the question, "What is an upset?" subjectively, we believe that this simple definition is too broad and includes matches that most people would not think are real upsets.

With that in mind, let's dig deeper ...


Perspective from March Madness

To frame the discussion, consider the first round games from the "March Madness" basketball tournaments. The tournament has four regions with 16 teams that are fully seeded - from #1 down to #16. There are first-round matchups between evenly-matched opponents (like when a #8 seed plays a #9 seed) and games intended to be lopsided (#1 vs. #16).

At one extreme, no #16 seed has ever beaten a #1 seed - the #1 seed is a perfect 128-0 in 1v16 matchups. When a #16 seed finally defeats a #1 seed, it will be big news - virtually everyone in the country will consider it to be a huge upset.

On the other hand, no one is surprised when a #9 seed beats a #8 seed. In fact the all-time win-loss record of #9 seeds against #8 seeds is 72-80 (47.4%). The #8s and #9s are evenly matched - and hardly anyone considers it an upset when the lower-seeded team wins.

What about something in the middle? Say, the #6 seed vs. the #11 seed? Most people would consider a win by the #11 seed to be an upset, but it is not all that uncommon. The #11 seeds are 52-96 all-time against #6 seeds, winning more than a third of the time. The #11 seed has had lots of success recently, winning five of the eight matchups with the #6 in 2015 and 2016. Since such upsets are common, many people spend time in March on their brackets guessing which double-digit seeds might advance.

In the NCAA basketball tournament, there are:

  • some matchups where there is a clear underdog,
  • some where you would flip a coin,
  • and some matches in between.

We will use these same three categories when we look at tennis ...


The Heavy Favorite Versus the Clear Underdog

Let's take a look at the tennis data we collected - starting with matches that have a clear underdog. The Tennis Recruiting rating system gives win probabilities for each matchup, and there are some matches where the favorite has a 90% chance or better to win. Most people agree that there is an upset when the underdog wins such a match.

Of the 20,000 matches in our studies, more than 6,000 fall into this "clear underdog" classification - where one player is expected to win 90-100% of the time. Looking specifically at these 6,641 matches, our system actually expected the favorite to win 96% of the time, and we call this figure the Expected Win Percentage or EWP. Here's how the various rating systems did for these 90-100% EWP matches:

EWP Range Matches How Often We Expected Favorite to Win How Often Favorite Actually Won
90‑100% 6,641 96.0% 95.4% 95.2% 89.3%

The table shows how the Tennis Recruiting rankings (TRN), Universal Tennis Ratings (UTR), and USTA Points-Per-Round (USTA) systems performed when predicting the winners of these matches. All three systems make similar predictions, and all three systems call the matches correctly the vast majority of the time.

Let's also take a look at the UTR definition of an upset. Universal Tennis considers a match an upset whenever a player defeats an opponent rated more than one full point higher on the UTR scale. We again think most people would agree that these are genuine upsets. Looking at the same data, 7,947 matches fall into this category. Take a look at how the three systems performed when predicting these matches:

UTR Difference Matches How Often We Expected Favorite to Win How Often Favorite Actually Won
> 1.0 7,947 92.5% 91.9% 91.7% 83.3%

Both the TRN and UTR systems predicted more than 91% of these matches correctly, and the USTA system chose the winner more than 83% of the time as well.

However you choose to slice the pie, wins by the lower-rated player in these situations don't come easy - or particularly often. When considering the total number of 20,000 matches, these upsets only occur about 1.5% of the time. There seems to be consensus that these wins are upsets.


Flip a Coin

So there is strong agreement on matches between players with very different skill levels, but what about players whose levels are very close? Let's consider the round of 64 match-up in the 2015 US Open between Steve Johnson and Thomaz Bellucci. Here is how the various systems rated these two players:

Steve Johnson 1,240 Points 15.44 51.3% EWP
Thomaz Bellucci 1,105 Points 15.52 48.7% EWP

While all three systems rate these players closely, they disagree. Johnson was rated higher by the ATP and TRN, but Bellucci was rated higher by Universal Tennis. So at least one of the systems was going to be wrong - independent of who won the match. Steve Johnson did end up winning, but almost no one would consider this result an upset. You might as well toss a coin. This scenario is very similar to the #8 versus #9 basketball games discussed above.

So let's take a look at all the matches where the higher rated player had a 50-60% chance of winning. A little over 3,000 matches fell into this category, and our EWP for these matches was only 55%.

EWP Range Matches How Often We Expected Favorite to Win How Often Favorite Actually Won
50-60% 3,038 55.0% 54.8% 53.8% 52.1%

None of the systems reached even this low 55% threshold. To put that into perspective, picking winners randomly would result in correct predictions about 50% of the time, so none of the systems did well for this pool of matches. Even though the lower rated player won almost half the time, not many would view these wins as upsets.

We see similar results if we look at players rated similarly by the UTR. The next table shows results for players rated within two-tenths of a point of each other on the UTR scale.

UTR Difference Matches How Often We Expected Favorite to Win How Often Favorite Actually Won
< 0.2 2,793 63.8% 60.9% 59.3% 55.8%

While this pool of matches does not seem to be quite as hard to call as the group listed above, none of the systems excelled at predicting the winners. Again, not many people would view these matches as upsets.


The Other Half

The above classifications by expected win percentage and UTR difference account for about half of the overall matches we examined. What about the other 10,000 matches or so in between? Do some of those matches fall into the upset category?

One of the best examples that a subscriber pointed out to us last year was Serena Williams' US Open semifinal match in her quest to achieve the calendar-year grand slam. Our model gave Williams a 86% chance of beating Roberta Vinci, and she was not rated a full UTR point above her opponent. This match does not fall into either the 90-100% EWP or >1.0 UTR categories listed above, but many viewed this win to be one of the biggest upsets in women's tennis history. So ... are the 90-100% EWP and >1.0 UTR classifications too exclusive?

Let's take a look at the data for these matches in the middle. First consider all the matches where the favorite has between a 60% and 90% chance of winning.

EWP Range Matches How Often We Expected Favorite to Win How Often Favorite Actually Won
60-90% 10,363 75.9% 73.9% 71.7% 63.4%

We expected that our algorithm would pick the correct winner 76% of the time, but notice that all of the systems fell a little short of this mark.

We also look at the numbers for matches between players with a UTR difference of between 0.2 and 1.0 points.

UTR Difference Matches How Often We Expected Favorite to Win How Often Favorite Actually Won
0.2-1.0 9,300 72.9% 71.5% 70.3% 63.0%

The results look similar to those from the 60-90% EWP group.

With all this in mind, what do we think? What is the best definition of an upset?


One Verdict

Many answers are possible when defining upsets, but if we have to pick one, we would probably settle on the following definitions:

An upset occurs when:

  • the winner has an EWP of 0-30% - or less than a 30% chance of winning the match.
  • the winner has a UTR more than 0.5 points lower than his/her opponent

Why 30%? That number expands the pool of potential upsets to more than 13,000 matches, and the lower-rated player wins just under 2,000 of these matches. With this definition, the total number of upsets goes to just under 10% of the 20,000 total matches.

Upset Definition Matches Upsets How Often Underdog Wins
(Subset) (Total)
< 30% EWP (TRN) 13,871 1,873 13.5% 9.3%
> 0.5 UTR Difference 13,334 1,954 14.7% 9.7%

An upset occurs often enough to be noticeable - but not so infrequently as to be a shooting star. Again, there is no real right or wrong answer to this question, but this our best attempt at an answer.

What do you think? Where do you draw the line between an unexpected win and an upset? We would love to hear from you in the comments below.

All the data used to compile the information in this article is available from our Analysis of Junior Rankings and Ratings.


Leave a Comment


More Ranking Articles

Behind the Tennis Recruiting Rankings
This week marks the 569th consecutive week that Tennis Recruiting has put out graduation-based rankings of American junior boys and girls. Rankings are front and center on Tennis Recruiting, and people often ask us how our rankings work. Today we describe our ranking system ...

An Overview of Ratings and Rankings
Tennis Recruiting is a website that rates and ranks junior tennis players, and because of that, we field many questions about how to interpret our lists. Questions like, "Are rankings better than ratings? Which is more important?" Or, "Since your system ranks by graduation year, are you able to compare players from different classes?" This article addresses the simple distinction between ratings and rankings.

Revisiting the US Open Forecast
The US Open is the premiere event on American soil, and so Tennis Recruiting pulled out its heat maps once again to forecast the tournament. There were lots of upsets (Wait, Serena lost?), but let's take a look at how our predictions played out...

 Ranking Article Index |   Subscribe

The lifeblood of Tennis Recruiting is its rankings - and our team has been ranking tennis players for more than two decades. Learn about tennis rankings in general - as well as our best-of-breed ranking system.
Recent Tournament Forecasts
Date   Tournament Predicted
8/29 U.S. Open 73.3%
8/05 USTA National Champs. 76.8%
7/17 USTA Clay Court Champs. 75.9%
7/02 Wimbledon 77.1%
5/29 Roland Garros 73.4%
Click here for the complete list of all of our Tournament Forecasts.
  Julia Goerges
Babolat PLAY Pure Drive
Was: $299.00  Now: $299.00
Review, Feedback, Demo, Video
  Garbine Muguruza
Babolat Pure Drive Wimbledon
Was: $209.00  Now: $179.00
Review, Feedback, Video
  Roberta Vinci
Head Graphene XT Extreme Pro
Was: $189.95  Now: $179.95
Review, Feedback, Demo, Video
  Elina Svitolina
Wilson Burn 100
Was: $199.00  Now: $99.00
Review, Feedback, Demo, Video
Ryan Harrison chats with fans 2013 US Open
Your calls. Our players. Totally live and unscripted. #CallASerialPlayer
3/29 University of Virginia's Brian Boland Succeeds Jay Berger as Head of USTA Men's Tennis
3/28 Bilokin Beats Bolton for Belated Birthday Gift; Top Seed Bryde Begins Quest for Adidas Easter Bowl Title with First Round Win; Semis Set for 12s and 14s
3/27 Former Champion Claire Liu Takes Wild Card into Adidas Easter Bowl ITF; Upsets in 12s and 16s Divisions; Team USA Developmental Coaches Awards Announced
3/26 Rotsaert Claims International ITF Grade 1 Spring Championships Boys Title, Branstine Sweeps Girls Titles; Easter Bowl Grade B1 Qualifying Complete
3/25 Johnson, Branstine Meet in International Spring Championships Girls Final; Ross and Rotsaert Play for Boys Championship; Woldeab Wins Epic 16s Boys Final
Colette Lewis has covered topflight junior events as a freelance journalist for over a decade. Read her weekly column, follow her on Twitter, and and find more of her daily commentary at ZooTennis.

Page updated on Sunday, February 12, 2017
Contact our web team with any corrections