We present here the overall performances of Kickoff.ai's model over the UEFA Euro 2016.
We published a paper! Find out about the model behind Kickoff.ai and its performances.
L. Maystre, V. Kristof, A. J. G. Ferrer, M. Grossglauser, The Player Kernel: Learning Team Strengths Based on Implicit Player Contributions, Machine Learning and Data Mining for Sports Analytics (MLSA), 2016.
We used two metrics to compare against some betting odds and a naive random predictor (it says I don't know all the time, using a probability of 50% for each team). As for now ties are not handled in our model, we discarded the 11 tied matches. The evaluation is hence done using 40 matches.
To compute the accuracy, we consider a correct prediction if the team with the highest winning probability according to a model actually wins the match. The higher the accuracy, the better the performances. With this scenario, Kickoff.ai guessed 26 matchs out of the 40. The accuracy is therefore 65%. As a comparison, the betting odds of some bookmaker companies obtained an average accuracy of 61.67% and choosing randomly one of the team gives 50%.
However, the core of our model lies in the fact that we estimate the probabilities of winning a match for each team. We believe that this probabilistic point of view is much more interesting, as it acknowledges that football is simply not always predictable. To take these probabilities into account, we use the complement of the Brier score. It is a fancy way of saying that we penalize predictions that are both confident and wrong. For instance, we were quite confident that Spain would win against Croatia, but the outcome was different so we made a big mistake. On the other hand, it was difficult to predict the outcome of Russia against Slovakia as the two teams have more or less the same strength, and our prediction was close for this one. Hence, we are not penalized too much by making a mistake on this match.
However, there is no direct interpretation behind the absolute numbers obtained by computing the Brier score. They make sense only when compared against each other. If we predict 50% for each team, then the score is 75%. Kickoff.ai's model obtains 77.55%, which is better than the random predictor, but worse than 78.39% obtained by the betting odds.
All in all, we are happy with our predictions! Our results are better than predicting each match randomly and close to the betting odds, which represent the wisdom of the crowd. Moreover, the model is actually quite simple in the sense that it considers the strength of a team as the sum of the players. This assumption is too weak, as a lot of factors play a role in the outcome of a game. We have plenty of ideas of improvements for our model, starting of course by predicting the ties. We believe that we only started to scratch the surface of the problem. If you are interested in future developments, do not hesitate to subscribe to our newsletter in the box below!
19 

S. Vokes 
9 

S. Church 
17 

G. Williams 
24 

D. Vaughan 
 

A. King 
20 

D. Cotterill 
11 

J. Williams 
 

D. Edwards 
15 

A. Richards 
 

J. Collins 
12 

O. Williams 
21 

D. Ward 
D. Origi 

17 
D. Mertens 

14 
C. Benteke 

20 
M. Batshuayi 

22 
M. Fellaini 

8 
C. Kabasele  –  18 
L. Ciman 

23 
J. Gillet 

13 
S. Mignolet 

12 
M. Dembélé 

 
T. Vermaelen 

3 
J. Vertonghen 

5 
 

T. Priškin 
17 

N. Nikolić 
13 

D. Böde 
 

Z. Stieber 
 

Á. Elek 
16 

Á. Pintér 
36 

M. Korhut 
35 

D. Dibusz 
22 

P. Gulácsi 
37 

B. Bese 
5 

A. Fiola 
 

K. Németh 
D. Origi 

17 
C. Benteke 

20 
M. Batshuayi 

22 
Y. Carrasco 

 
M. Dembélé 

 
J. Lukaku 

21 
C. Kabasele  –  18 
J. Denayer 

15 
J. Gillet 

13 
S. Mignolet 

12 
L. Ciman 

23 
M. Fellaini 

8 
 

P. Carlgren 
 

R. Olsen 
 

L. Augustinsson 
13 

P. Jansson 
2 

M. Lustig 
21 

J. Durmaz 
15 

O. Hiljemark 
 

O. Lewicki 
16 

P. Wernbloom 
22 

E. Zengin 
20 

J. Guidetti 
 

E. Kujović 
J. Gillet 

13 
S. Mignolet 

12 
L. Ciman 

23 
J. Denayer 

15 
C. Kabasele  –  18 
J. Lukaku 

21 
M. Dembélé 

 
M. Fellaini 

8 
M. Batshuayi 

22 
C. Benteke 

20 
D. Mertens 

14 
D. Origi 

17 