When Bias Emerges

Let’s look at

The points at which bias is introduced

Training Data

Training data is the most commonly cited source of algorithmic bias. When the data used to train models is unrepresentative of the population that the model will ultimately serve, models will perform poorly for those groups left out of the data. For example, facial recognition technologies that perform poorly on female faces with dark skin do so, in part, because the dataset of faces that trained the model did not contain enough dark-skinned female faces. Note that Black women's exclusion in large face datasets reflects societal biases against Black women that also manifest in Black women's underrepresentation in movies, TV, and marketing, or the limited selection of cosmetics made for Black complexions and kinky hair.

Training data can contain artifacts of societal biases even when protected groups are overrepresented in the data. Frequently, the ground truth outcome variable we are trying to predict is just a representation of the values human decision-makers held before algorithmic tools stepped in. Tools like COMPAS (Correctional Offender Management Profiling for Alternative Sanctions), which claim to assist judges with assessing how likely defendants are to become recidivists, rely on historical data to make their predictions. These data are a log of decisions made by socially-powerful communities (i.e., judges, lawmakers, and law enforcement) over socially marginalized communities (i.e., those disproportionately represented in the criminal justice system). As such, they contain artifacts of the values and beliefs of those decision-makers with little input from those who have faced sentencing, over-policing, and various social barriers to community safety. Journalistic investigation of COMPAS found that the tool produced racially-biased predictions. These biases reflect values about race, gender, and criminality in the US carceral system. They are, in many ways, a product of American chattel slavery and European colonization.

Training datasets do not exist in a vacuum. Rarely, if ever, are they immune to the inequities that permeate our world.

Model Selection

AI's ability to work well on any given task is almost entirely dependent on model selection.

Simpler models that are easy to troubleshoot are rarely the most accurate. More sophisticated models open the doors to all kinds of bugs, misinterpretations, and errors. While an experienced programmer may be unlikely to introduce bugs into a pre-packaged solution like SciKit Learn, complex models like Bayesian Neural Networks with Hamiltonian Monte Carlo Sampling almost always require hand-coding and, therefore, lots of debugging.

Inadequate Objective Functions

In most machine learning applications, we create objective functions that allow us to maximize accuracy. For example, in an algorithm that sets prices for e-commerce products, the objective function may be to get as close to the true price a customer is willing to pay for a product. That algorithm wants to minimize the money left on the table as well as the risk that the customer will find the price too high and leave. However, accuracy is not the only, or even the most important, metric to maximize in many cases. While it may be the case that we want to predict which YouTube video will be most engaging for a viewer to watch, we should not maximize engagement without consideration for the promotion of hate speech. In the case of mitigating radicalization pipelines on YouTube, for example, optimization functions must consider more than just view time—they should also consider user behavior, sentiment, and likelihood to engage in violent speech on the platform when making recommendations.

Many algorithms fail by having too narrow of an objective, whether that be clicks, views, or some other simple metric that is easy to measure in the short term. While it is much more challenging to account for long-term, downstream effects and incorporate them into predictive models, we must begin to shift our thinking in this direction. Researchers have begun to make compelling arguments for why we need a developmental approach to understanding algorithmic impact. The more we interact with predictive technology, the more it will affect our evolution as individuals and as a species.

Failure Modes

Many machine learning algorithms suffer from too little experimentation before deployment. When algorithms "break" in the real world or cause adverse harm, it is said to be an unfortunate but rare oversight rather than a symptom of an inherent flaw in the development process. This simply isn't true. We must hold developers accountable for their missteps much in the same way that we hold doctors and lawyers accountable for malpractice. Just like doctors and lawyers, machine learning engineers hold people's lives in their hands.

A popular ethos within the tech world is to "move fast and break things." And in fact, many models are deployed without sufficient investigation into failure modes. Just because a model performs well in an experimental setting with a toy dataset does not mean that it will work well in the wild. You wouldn't practice flying a remote control airplane and then conclude you're prepared to fly a 747, right? It is impossible to anticipate all the ways an algorithm may harm individuals or communities. However, marginalized communities are rarely front-of-mind during development projects. This is particularly evident in the lack of community engagement around development projects (i.e., asking vulnerable communities what products they would like to see built) and lack of diversity among engineering staff. Engineers—who are overwhelmingly white, male, cisgender, and upper-middle class—largely build products for themselves and their communities. Should we really trust people who fail to hire employees in a way that's nondiscriminatory to innovate in ways that are nondiscriminatory? Probably not.

Deployment

The ways that machine learning systems are used in the world provide even more avenues for harmful effects. Often, models that are developed for one context are then lifted and used in others for which the developers did not intend. This is known as transfer context bias. Additionally, human users may receive algorithmic predictions but then misinterpret or misuse those findings. Commonly, a lack of understanding of how prediction works leads non-technical users to assume that a machine learning system is objective and precise, when in fact many predictions contain a high degree of uncertainty. This is known as interpretation bias.

White Supremacy in the Tech World

It is important to acknowledge that the very foundation upon which the tech giants rest is one of white supremacy. Silicon Valley, and the academic institutions that birthed it, would not exist without generations of wealth extraction and accumulation via colonization, slavery, segregation, and imperialism. It should come as no surprise, then, that the people who fund the world's most powerful products are 64% white men. In 2018, only 4% of individuals ranked in the top 100 VC partners were women of color and none were Black women. A handful of individual investors' values largely shape which technologies get to be created, by whom, and how decisions are made along the way. Both the explicit and implicit biases of technologists, a relatively homogenous group, undergird the products that power our world. This is not just about the galling lack of diversity at VC firms. It is also about the fact that the people in Western, educated, industrialized, rich and democratic (WEIRD) societies only account for 12% of the world's population. The most wealthy among them? Well, they only account for a small fraction of 1% of the world's population. To say that those individuals may be out of touch with the needs of the majority of humanity would be an understatement.