Today, there was an article posted (https://blogs.magicjudges.org/road-to-l3/2018/05/23/gender-bias-in-the-l3-advancement-process/) that analyzes data gathered from L3 candidates over the past several years. The article examines potential bias against female candidates and concludes that there is no evident bias. When reviewing the data from the article, I saw a different story that is in my opinion more consistent with what I have seen. Like the conclusion in the article, this is just a hypothesis, but I think it is an alternative view worth sharing.
Analysis of L3 Panel Data
The articles notes that women are more likely to pursue L3, with 10.2% of the panels in the data being female candidates, compared to 7% of L2s being women. Here, I would note that data is being used from 2012 to 2017, which is a very long range in terms of evolution of the judge program. I would be curious to see how that percentage is weighted over the years. If most of the women in this sample paneled in the past couple years, then it would make sense that this percentage might be “ahead” of the L2 data, because there are many, many judges who get to L2 and stay there. L3 candidates are, to my knowledge, more likely to be more recent judges, so the percentage of L3 candidates that are female may be closer to the percentage of L2s certified in the past 5 years who are female than it is to the percentage of female L2s overall. This is a potential explanation for this result.
If it really is the case that female judges are more likely to try for L3, another potential explanation is that female judges get a lot more attention, both positive and negative, than male judges on average. Since there are relatively few female judges, they tend to stand out more, resulting in increased visibility and feedback. Not all of that is positive for an L3 attempt, as negative feedback is also more prevalent and mistakes are remembered more, but this might be a contributing factor.
Here it appears that, with a few exceptions, men and women fare about equally in terms of deficiencies. In Leadership, Presence, and Charisma and very substantially in Penalty and Policy Philosophy, women show deficiencies more often than men. There are several areas where men have shown deficiencies while female candidates have not shown any, but many of these are old qualities that are no longer scored, which calls further into question how many female candidates were evaluated while those qualities were considered. Of modern qualities, male candidates showed deficiencies in self-evaluation and in Teamwork, Diplomacy and Maturity while women did not.
The article concludes that gender bias and stereotypes do not play a role here, because if they did, then Stress and Conflict Management would show a bias against female judges too. I think there are multiple different stereotypes and biases at play here, and this warrants some further investigation.
Leadership, Presence, and Charisma and Penalty and Policy Philosophy are both qualities that, in terms of their evaluation, involve authority. Authority is obviously involved in the former, and in the latter, authority is what makes others, including evaluators, take your opinions and conclusions seriously. In speaking with other female judges, I’ve heard again and again how difficult it is to get others to take you seriously. From players asking for “a real judge” or not believing rulings to other judges explaining simple tasks to female judges who have done them dozens of times, walking into a group of strangers and being taken seriously as an authority is often a problem for female judges. There is evidence in many fields (https://www.theguardian.com/books/2015/aug/06/catherine-nichols-female-author-male-pseudonym, https://link.springer.com/article/10.1007/s11199-016-0586-1) of similar problems.
Stress and Conflict Management falls under a different stereotype, where women are considered less tough than men in similar situations, but I would argue that this is balanced by the stereotype that women are “nicer” and less likely to cause or escalate conflict in the first place. That, combined with the fact that there is a lot of social conditioning that has caused female judges to typically be less aggressive and more friendly in their interactions, makes me skeptical that a lack of demonstrable difference in pass rate rules out gender bias. It would be illuminating to see the general reason this quality was failed by male vs. female candidates, but with so little data, that is likely not sufficiently anonymized to publish.
Analysis of Questionnaire Data
Here, the article concludes that women are more amibitious and further along in the L3 process than male candidates. I would argue that an alternative conclusion that can be drawn from the same data is that women are further along in their judging career and in their process before they consider themselves L3 candidates. The distribution and advertisement of this questionnaire was essentially to judges who were L3 candidates or former L3 candidates, so the folks answering these questions considered this relevant to their interests. For men, a large percentage of respondants were not really interested in L3, but considered their opinions relevant enough to answer the survey. Women, by percentage, appear more likely to not respond unless they were already seriously involved in the L3 path. Rather than showing that more women pursue L3, it is possible that women are more conservative in considering themselves potential L3 candidates.
In terms of encouragement, a similar picture appears. The article draws the conclusion that female candidates are more encouraged to pursue L3, but that data could just as easily mean that women wait longer in their judge careers and are stronger judges before pursuing L3, or that women need more encouragement to feel ready to pursue L3. It also could speak to a higher percentage of female respondants being more advanced in their process than men taking the same survey. Many of the “no interest in L3” men who took the survey are possibly not ready for L3 and therefore wouldn’t have gotten encouragement towards it.
Perception of Bias
Here, it is overwhelmingly the case that women feel that they are treated and evaluated differently based on their gender. The article says that women feeling that the L3 process is harder for them is at odds with them receiving more encouragement, but this is far less contradictory if I am correct about the underlying selection bias in this data. In any case, the fact that there is a trend that men feel the L3 process is slightly easier for them and women think it is slightly harder for them speaks to a consistent problem at least in perception and perhaps in process that needs to be taken seriously and addressed.
I don’t feel that the conclusion of the article, that there is no bias evident and that women receive more encouragement but feel that the L3 process is harder for them, is appropriate to the data. The confounding variable of selection bias is a major issue here, and as I have outlined, could very easily explain some of the results that the article found to be contradictory. My hypothesis, based on this data and my experience, is that women are less likely to consider themselves L3 candidates until they are further along in their judging career and in the process than men. That also means that women were less likely to respond to this survey unless they were further along in the process, because that’s what it took for them to feel that this was relevant to them as L3 candidates. There is some evidence that gender bias, particularly surrounding authority, could have an impact on deficiencies, but frankly, there isn’t enough good data to draw a real conclusion about that, especially given the change in L3 qualities.
My conclusion is that while there are some interesting trends in the data, there is very little conclusion to be drawn here, and stating that there is no evident bias is too strong. The only conclusion that I think is backed up by the evidence is that women feel that there is unfairness in this process, and to me, that means a qualitative consideration of the process is probably needed to examine that perception. Ultimately, saying that women are just perceiving the unfairness and are being encouraged plenty is what I think a lot of people will get from the article, and that’s not a fair conclusion based on the data.