Why Gendered AI Features are a Problem for Tech Companies

On May 30th, CrowdStrike launched "Charlotte AI," the company's new #generativeAI "security analyst" for its platform. Like #Microsoft, #SentinelOne, and #Google, this LLM-based feature will allow users to "ask natural language questions – in English and dozens of languages – and receive intuitive answers." However, unlike the others, CrowdStrike introduced its feature with a gendered name and visual persona, "Charlotte," a character first introduced in the company's February "Troy" ad.

Here I will argue that this product decision is highly problematic on two fronts: reinforcing human biases and obscuring technological understanding at a time when clarity is needed.

The incorporation of LLMs into security tooling has great upside. As all of the aforementioned companies have argued, natural language interfaces can, among other things:

• Dramatically lower the barrier to entry by reducing reliance on scripting and querying. Being able to "converse" with platforms can reduce the cognitive load for analysts and, more importantly, reduce MTTR.
• Free up limited human resources to focus on larger issues with remediation, by offloading and automating more mundane tasks where computational intelligence is faster than human cognition.

However, labeling #machinelearning capabilities with gendered human naming conventions has been demonstrated to exacerbate human harms and obscure understanding.

Gendered AI Reinforces Human Bias & Stereotypes

In a 2019 UNESCO study, "I'd Blush If I Could," the organization looked at the rise of gendered AI applications, in the form of virtual assistants like Alexa, Cortana, Siri, and Google Assistant, all of which were given specifically designed female voices and imagined personas in the design process. The study looks at how human stereotyping of women as subservient is reinforced by these technologies because “it sends a signal that women are ... docile and eager-to-please helpers, available at the touch of a button."

Of course, there's a distinction between verbal commands versus those fed into a LLM on a UI level. But humans are context-making creatures, so it's easy to see how delegating "repetitive and tedious tasks like data collection, extraction, and basic threat search and detection" (emphasis mine) to a product feature gendered as a woman reinforces notions of what women in tech are qualified to do.

More to the point, the UI demonstrated in CrowdStrike's blog post further personifies the processing of requests with the clear image of Charlotte AI as a woman.

It doesn't take a leap of imagination to see how soon analysts assigned with "boring" tasks in a SOC might find themselves saying, "Just make Charlotte do it."

"What's in a name?" - Romeo and Juliet, Act II, Scene II

These product decisions do not occur in a vacuum. On the one hand, "Charlotte AI" might be seen as empowering, by placing a woman as central to new, transformative detection capabilities. But the reality of women's experiences in the tech sector tells a different story. Women remain conspicuously absent in #cybersecurity leadership roles. In a November 2022 survey of 1500 women working in tech, nearly half cited an increase in workplace sexual harassment. In the US alone, "more than 1 in 5 women in tech have experienced verbal abuse, sexual harassment or intimidation in the workplace."

Gendering product features that automate lower skill tasks can influence the how that gender is perceived in the real world. It's absolutely unnecessary to assign gender to product features. One need go no further than to ask female or non-binary colleagues about abusive or toxic language they've seen in Discords or other cybersecurity discussion forums.

From a UX standpoint, asking a "co-pilot" or querying a "workbench" are both more neutral and a more accurate representation of the technology. Gendering exacerbates the continued problem of anthropomorphizing #artificialintelligence processes.

AI is Not Human or Magic

LLM interfaces are powerful tools, but they form part of a process not an embodied entity. They do not "know" or "think" in the way human counterparts on a security team do. To ascribe human personalities to #machinelearning features is a dangerous obfuscation of how the technology operates. In a 2021 blog post for the Brookings Institution, mechanical engineering professor Cindy M. Grimm, argues that "correct use of mechanistic, pedantic language is a powerful tool to reveal...components and correctly bracket their capabilities."

As more LLM tooling comes online, it's important to ensure the cybersecurity workforce is up-skilling appropriately. If analysts do not understand how LLMs work, they cannot understand their limitations and where more human intervention may be necessary during investigations or incident response.

AI literacy will become necessary for every citizen, but it is imperative for tech workers to gain this literacy now. Personifying and gendering processes gets in the way of that goal, especially when AI researchers and developers themselves are struggling to explain how their own technology works.

Conclusion

To be clear, CrowdStrike's capabilities or the performance of their solutions are not in question here. Instead, I wanted to simply call out why naming the company's generative AI feature "Charlotte AI" poses social and educational risks. There is marketing value and a visual storyline to be sure. However, against the backdrop of misogyny and under-representation in the tech sector, alongside issues of ethical AI application, the choice is confounding and problematic. We know and have known of the ills associated with gendered technology, arguably since "Rosie" the robot on The Jetsons. Let us think more deeply about these new technologies, and let's try to get it right.