MMR and Who Actually Wins
Here's something we see a lot in CEA.
It makes sense. As humans, we have a perpensity to make metrics out of things with numbers, and hand-wave the rest as details. Compare the numbers, find the winner. Sports are a solved problem. Right?
First, let's define some terms. Modern games that function as esports do a good job of telling their players how good they are compared to other players. Your in-game skill rating is referred to by a number of different titles: Match Making Ranking (MMR), Rank, Skill Rating (SR), etc. For our purposes, these all mean the same thing - how good the game *thinks* you are. That leads us to our question: When the game says you are better, do you actually win the match?
There is certainly correlation between MMR and match wins. It's easy to start with the theory "The team with the better MMR will always win" and see where it takes you. It took us to an analysis of 153 team-based games played in our Fall 2020 CEA season. These were games where we had player-specific MMR and were able to compare it to the end result of the match.
We found that the team with the higher MMR won a whopping... *drumroll please!... **72%** of the time.
Oof. That seemed like a big discrepency. In 153 matches, 42 were what we call "upsets" - games where the higher ranked team did not come out ahead. That much difference meant our MMR-based model was closer in accuracy to a coin flip than total success. We needed a better explanation.
The first major realization was that almost never did a team underperform their MMR one week and then overperform the next. Of the 42 upsets, 5 teams accounted for more than half of the losses. These were high to mid-range MMR teams that by our MMR-based model should have finished the season heaped in glory. Needless to say, they did not.
The upset wins, however, were more well-distributed across a range of MMRs. To be clear, in no case did a bronze-ranked team take down a group of diamonds, but upsets frequently crossed large swaths of rank differential. Teams that might have been forgiven for writing off their match as neigh-impossible were as likely to score an upset win as teams relatively tight in MMR.
That was the first takeaway: MMR doesn't give you a guarenteed win - it just makes it your match to lose.
But we also wanted to understand what happened on teams that were rock solid - highly ranked teams with no upset losses. We asked them what, in their mind, made them successful.
Teams with one or more upset win often:
- Describe themselves as friends
- Communicate during times not set aside for practice or official matches
- Enter matches with a plan
This brings us to a new theory: The 28% is quite real - it's communication, morale, and team-based play. Players that want to identify as part of that team are more likely to make that team succeed.
Of course, how could anyone believe this? If you wanted to put metrics around any of it you would have to be able to measure the way teams talk to one another, and then distinguish good communication from bad communication. To really confirm this theory, we would have to listen to teams talk across a variety of games, grade their communication, and compare it to their results.
So check back next time, when we do exactly that.
