Judging Correlation in the Mundial de Tango (more Fun with Data)

Ok, in the previous post I said that there's no agreement among the judges during the final of the Mundial de Tango (Pista) or Tango World Championship. I showed you the charts that convinced me, but I didn't properly measure and show the degree of disagreement.

This second Power BI report has a page for each year available. You can select any two individual judges and see how far they agreed with each other about how to rank the couples. So, if you think, for example, that two of the judges dance a similar style to each other, you can see* if their opinions about the finalists' quality of dance correlate with each other. (Spoiler: nope.)**

To change the year, move to the next page using the arrows at the bottom centre. If the report is too small, misbehaves, or won't fit on your screen properly, try popping it out with the diagonal arrow thing at the bottom right hand corner. You might have to scroll the selectors right and left to see all the judges.


The judges' rankings of the couples do not correlate with one another.

1.00 is a perfect correlation: each judge agrees perfectly with him-or-her self. A low correlation between two judges means they didn't agree much, and a negative correlation would mean they ranked the couples in the opposite way to each other. There are one or two cases of small negative correlations.

I'm sure all the judges' opinions on people's dancing, in various circumstances, are highly valuable - that's why they were picked to judge - but they have nothing to do with one another, and their collective decisions are therefore, to put it mildly, not much help to anyone else in distinguishing between the finalists.

One reasonable interpretation of this result is that the judges have an impossible task; all the couples in the final dance in much the same way, and there is no real difference between them that the judges could possibly agree about. It is as though you, I, and five of our mates solemnly and conscientiously gave scores to the aesthetic qualities of six eggs from the same nest.

Why are the eggs all from the same nest? Perhaps because any excellent dancer with a visually-apparent difference of style and musicality would, on the face of it, have much to lose and nothing to gain by entering this competition. But even if the dancers were different, while all good, it's not clear that would help; it might be even more meaningless to decide between them.

There may be different interpretations: go ahead and put them in the comments, and let's see if we can think of a way to tell which is right. One would be that there are real differences, but the judges don't agree about which ones are important; they are using totally individual and independent criteria. No information is published about what criteria they use.

In order to distinguish between the couples, the judges would have to agree both on what differences exist and on which ones are important. For example, because of the way the couples get to the final, one of them is usually much older and less mobile than the others. It seems to me that the judges have agreed that the differences which go with that are not important, although I don't have the couple-number data to show that; the only way to get it is to watch the video.

As for what it means, and whether it is a good thing, we began to talk about this in the comments on the previous post.

I think it is a good thing that the Mundial is not like a ballroom competition, with the rigidity and the arms-race that implies; that could be very toxic for something that wants to remain a living social dance.  I don't think that finding the best dancers out of a good bunch is what the Mundial is really for. As I said before, it makes more sense to think that its purpose is to bring a steady stream of decent young salonsters to public notice, while honouring the occasional veteran; it's a very pretty industry-promotion and heritage-publicity thingie, not a sport.

Indeed, perhaps the Mundial has a somewhat paradoxical role in protecting tango from ballroomisation. All the finalists indisputably have good looking technique, whereas there are ballroom schools teaching a genuinely ballroomised argentine tango with a totally different  technique and approach, completely clueless about the social scene. The international dance associations even include it in some of their competitions (and that, for UK readers, was what Vincent and Flavia were up to with their "Tango World Champions" thing, which I've explained elsewhere). We can fairly confidently say that nobody dancing that way would ever get to the Mundial final, at least not in the Pista category - and that is a good thing. It's good that the Mundial exists and people can discover, quite easily, that the ballroom competitions are not it. But the relationship between regular ballroom schools, various international dance organisations, and Argentine tango, is another interesting subject for further research.

It would be great to have judge-level scores from earlier rounds. I'd expect to see a lot more agreement at the lower end; if we could combine that with video, we'd be able to learn something about what criteria are really being applied. And, if so, I'd expect to find that those criteria are by their nature useless in the final. Unfortunately, that data isn't published. If you think you can obtain it, please comment.

Bottom line: there's no evidence here that there's any point in remembering who won.

------------------
*You'll notice some straight vertical and horizontal lines in the charts. Judges rank a lot of couples equal with one another. They don't give forty different marks to forty different couples. I haven't done the calculations over the marks seperately from the rankings; I thought doing rankings would be clearer, as the judges don't work around any common average. Some judges give out marks only from a restricted set of integers, but others try to make fine distinctions. They see each couple dance three tracks. The see them in groups of ten to a dozen couples, and the couples don't all dance the same tracks - have a look at the post on Music in the Mundial for a description of the procedure, and links to video. 
** To be fair, there is one case of a nearly 0.7 correlation, which is very impressive compared with all the others, and you probably could say the two judges involved went together. I won't spoil that one, as it would be much better if you tried to predict who it would be and then looked. Maybe it's real, or maybe it just had to happen accidentally somewhere. There are also some cases of unimpressive 0.3 or 0.4 correlations looking strong against a background of zero to negative correlations. People who are personally acquainted with the judges might feel there was something to say there, but I'm sceptical that it isn't pure chance.