We present here the overall performances of Kickoff.ai's model over the UEFA Euro 2016.
Kickoff.ai's performances for the Euro 2016. Accuracy and Brier score are explained bellow. Higher numbers mean better performances. Click on the legend to show/hide metrics.
We published a paper! Find out about the model behind Kickoff.ai and its performances.
L. Maystre, V. Kristof, A. J. G. Ferrer, M. Grossglauser, The Player Kernel: Learning Team Strengths Based on Implicit Player Contributions, Machine Learning and Data Mining for Sports Analytics (MLSA), 2016.
We used two metrics to compare against some betting odds and a naive random predictor (it says I don't know all the time, using a probability of 50% for each team). As for now ties are not handled in our model, we discarded the 11 tied matches. The evaluation is hence done using 40 matches.
To compute the accuracy, we consider a correct prediction if the team with the highest winning probability according to a model actually wins the match. The higher the accuracy, the better the performances. With this scenario, Kickoff.ai guessed 26 matchs out of the 40. The accuracy is therefore 65%. As a comparison, the betting odds of some bookmaker companies obtained an average accuracy of 61.67% and choosing randomly one of the team gives 50%.
However, the core of our model lies in the fact that we estimate the probabilities of winning a match for each team. We believe that this probabilistic point of view is much more interesting, as it acknowledges that football is simply not always predictable. To take these probabilities into account, we use the complement of the Brier score. It is a fancy way of saying that we penalize predictions that are both confident and wrong. For instance, we were quite confident that Spain would win against Croatia, but the outcome was different so we made a big mistake. On the other hand, it was difficult to predict the outcome of Russia against Slovakia as the two teams have more or less the same strength, and our prediction was close for this one. Hence, we are not penalized too much by making a mistake on this match.
However, there is no direct interpretation behind the absolute numbers obtained by computing the Brier score. They make sense only when compared against each other. If we predict 50% for each team, then the score is 75%. Kickoff.ai's model obtains 77.55%, which is better than the random predictor, but worse than 78.39% obtained by the betting odds.
All in all, we are happy with our predictions! Our results are better than predicting each match randomly and close to the betting odds, which represent the wisdom of the crowd. Moreover, the model is actually quite simple in the sense that it considers the strength of a team as the sum of the players. This assumption is too weak, as a lot of factors play a role in the outcome of a game. We have plenty of ideas of improvements for our model, starting of course by predicting the ties. We believe that we only started to scratch the surface of the problem. If you are interested in future developments, do not hesitate to subscribe to our newsletter in the box below!
| 19 |
|
S. Vokes |
| 9 |
|
S. Church |
| 17 |
|
G. Williams |
| 24 |
|
D. Vaughan |
| - |
|
A. King |
| 20 |
|
D. Cotterill |
| 11 |
|
J. Williams |
| - |
|
D. Edwards |
| 15 |
|
A. Richards |
| - |
|
J. Collins |
| 12 |
|
O. Williams |
| 21 |
|
D. Ward |
| D. Origi |
|
17 |
| D. Mertens |
|
14 |
| C. Benteke |
|
20 |
| M. Batshuayi |
|
22 |
| M. Fellaini |
|
8 |
| C. Kabasele | – | 18 |
| L. Ciman |
|
23 |
| J. Gillet |
|
13 |
| S. Mignolet |
|
12 |
| M. Dembélé |
|
- |
| T. Vermaelen |
|
3 |
| J. Vertonghen |
|
5 |
| - |
|
T. Priškin |
| 17 |
|
N. Nikolić |
| 13 |
|
D. Böde |
| - |
|
Z. Stieber |
| - |
|
Á. Elek |
| 16 |
|
Á. Pintér |
| 36 |
|
M. Korhut |
| 35 |
|
D. Dibusz |
| 22 |
|
P. Gulácsi |
| 37 |
|
B. Bese |
| 5 |
|
A. Fiola |
| - |
|
K. Németh |
| D. Origi |
|
17 |
| C. Benteke |
|
20 |
| M. Batshuayi |
|
22 |
| Y. Carrasco |
|
- |
| M. Dembélé |
|
- |
| J. Lukaku |
|
21 |
| C. Kabasele | – | 18 |
| J. Denayer |
|
15 |
| J. Gillet |
|
13 |
| S. Mignolet |
|
12 |
| L. Ciman |
|
23 |
| M. Fellaini |
|
8 |
| - |
|
P. Carlgren |
| - |
|
R. Olsen |
| - |
|
L. Augustinsson |
| 13 |
|
P. Jansson |
| 2 |
|
M. Lustig |
| 21 |
|
J. Durmaz |
| 15 |
|
O. Hiljemark |
| - |
|
O. Lewicki |
| 16 |
|
P. Wernbloom |
| 22 |
|
E. Zengin |
| 20 |
|
J. Guidetti |
| - |
|
E. Kujović |
| J. Gillet |
|
13 |
| S. Mignolet |
|
12 |
| L. Ciman |
|
23 |
| J. Denayer |
|
15 |
| C. Kabasele | – | 18 |
| J. Lukaku |
|
21 |
| M. Dembélé |
|
- |
| M. Fellaini |
|
8 |
| M. Batshuayi |
|
22 |
| C. Benteke |
|
20 |
| D. Mertens |
|
14 |
| D. Origi |
|
17 |