[MUSIC PLAYING]
AUDACE NAKESHIMANA: In our work on fairness and AI,
we present a case study on natural language processing
titled "Identifying and Mitigating
Unintended Demographic Bias in Machine Learning."
We will break down what each part of the title means.
This is the work that was done jointly
by Chris Sweeney and Maryam Najafian.
My name is Audace Nakeshimana.
I am a Researcher at MIT, and I'll be presenting their work.
The content of the slides presents a high-level overview
of a thesis project that was done throughout a course
of the year.
It will be released soon on MIT DSpace.
AI has the power to impact society
in a vast amount of ways.
For example, in the banking industry,
many companies are trying to use machine learning to figure out
if someone will default on a loan given the data about them.
Now, because machine learning is used
in the high-stakes applications, errors
that cause it to be unfair could cause discrimination,
preventing certain demographic groups from gaining access
to fair loans.
This problem is especially important
to address in developing nations where
there may not be existing sophisticated credit systems.
Those nations will have to rely on machine learning models
to make these high-stakes decisions, such as alternative
credit scoring mechanisms that are possibly going to be
involving AI more and more.
This work focuses on applications
of machine learning in natural language processing.
NLP is important to studying fairness
in AI because it is used in many different domains,
from education to marketing.
Furthermore, there are many sources
of unintended demographic bias in the standard natural
language processing pipeline.
Here we define the NLP pipeline as a combination
of steps involved, from collecting natural language
data to making decisions based on the NLP models trend
and resulting data.
Lastly, [? therefore, ?] natural language processing systems
is easier to get.
Unlike tabular systems from banking or health care,
where companies may be reluctant to release data
due to privacy concerns, NLP data,
especially in widely spoken languages like French
or English, is available from different sources, including
social media and different forms of formal and informal
publications, making it more effective to use in research
on how to make NLP systems more fair.
We now break down what unintended demographic bias
means.
The unintended part means that this bias
comes as an adverse side effect, not deliberately learned
in a machine learning model.
The demographic part means that the bias translates
into some sort of inequality between demographic groups
that could cause discrimination in a downstream machine
learning model.
And finally, bias is an artifact of natural language processing
pipeline that causes this unfairness.
Bias is [INAUDIBLE] term.
Therefore, it is important that we
center on a specific form of bias that
causes unfairness in typical machine learning applications.
In gender-based demographic bias,
for example, machine learning model
might associate specific types of jobs to specific gender
just because it's the way it is in the data used
to train the model.
Within unintended demographic bias,
there are two different types of bias
that will focus on in natural language processing
applications.
These are bias in sentiment analysis systems
that analyze positive or negative feelings associated
with words or phrases and toxicity analysis
systems designed to detect derogatory or offensive terms
in words or phrases.
Sentiment bias refers to an artifact
of the machine learning pipeline that causes unfairness
in sentiment analysis systems.
And toxicity bias is an artifact of the pipeline that
causes unfairness in systems that tries to predict toxicity
from text.
In either sentiment analysis or toxicity prediction,
it is important that our machine learning model
doesn't use sensitive attributes describing
someone's demographic to inform them whether a sentence should
be positive or negative sentiment or toxic
or less toxic.
Toxicity classification is used in a wide variety
of applications.
For example, it can be used to censor online comments that
are too toxic or offensive.
Unfortunately, these algorithms can be very unfair.
For example, the decision of whether sentence
is toxic or non-toxic can depend solely
on the demographic identity term,
such as American or Mexican, that appears in the sentence.
This unfairness can be caused by many different artifacts
of the natural language processing pipeline.
For instance, certain nationalities and ethnic groups
are specifically more frequently marginalized.
And this is reflected in the language usually associated
with them.
Therefore, training NLP algorithms and resulting data
sets could result in a certain form
of unintended demographic bias.
We want to drive home the point of unintended demographic bias
versus unfairness.
Unintended demographic bias can enter a typical machine
learning pipeline from a wide variety of sources,
from the word corpus to the word embedding,
the data sent to the algorithm, and finally from the thresholds
used to make decisions.
The possible unfairness or the discrimination
comes at the point where this machine learning
model meets society and actually causes harm.
This work addresses mitigating and identifying
unintended demographic bias at each stage
in the natural language processing pipeline,
from the words corpus to the decision level.
Our big goal here is to find ways
to mitigate the bias that we might inherently
find in the text corpora or other types of data
representation that are used to build NLP applications.
For this module, we cover measuring
unintended demographic bias in word embeddings
and using adversarial learning to mitigate word embedding
bias.
The corresponding thesis goes further,
and it covers techniques for identifying and mitigating
unintended demographic bias at other stages of the NLP
pipeline.
We now cover the work as measuring word embedding bias.
Word embeddings [? encode ?] [? text ?] into vector spaces
where distances between words describe a certain semantic
meaning.
This allows one to complete the analogy of man
is to woman as king is to queen.
Unfortunately, researchers Tolga Balukbasi and others
found that even for word embeddings trained from Google
News articles, there exists bias in word embedding space, where
the analogy becomes man is to woman as computer programmer is
to homemaker, another word for a housewife.
This is concerning given that word embeddings could
be used in natural language processing applications devoted
to predicting whether someone should get a certain job.
However, it is difficult to quantify the bias just based
on the vector space analogies.
In this work, researchers Sweeney and Najafian
develop a system to measure sentiment bias
in word embeddings to a specific number.
The way they do this is they take the bias toward embeddings
and use them to initialize an unbiased
labeled word sentiments data set.
They train a logistic regression classifier on this data set,
and they predict negative sentiment
for a set of identity terms.
For example, in this case, this is
a set of identity terms describing demographics
from different national origins.
They analyzed the negative sentiment
for each identity term and predict
a score that describes the bias in word embeddings.
This score is the divergence between the [INAUDIBLE]
for abilities, for negative sentiment,
for national origin, identity terms,
and the uniform distribution.
The uniform distribution describes a perfectly fair
case, wherein a demographic is receiving
an equal amount of sentiment in the word embedding model.
Now that we have a grasp on the word embedding bias,
we can start to figure out how to mitigate some of this bias.
In the thesis, Sweeney and Nafajian
describe how they use adversarial learning
to debias word embeddings.
Different identity terms can be more or less correlated
with positive or negative sentiment.
For example, words like American, Mexican, and German
can have more correlations with negative sentiment subspaces
and positive sentiment subspaces,
because in the data sets used, it
might appear to be more frequently associated
with negative or positive sentiments.
This is concerning.
Even downstream machine learning model
picks up on these correlations.
Ideally, you want to have each of those identity terms
to a neutral point between negative and positive sentiment
subspaces without distorting their meaning within the vector
space so that the word embedding model can still be useful.
They use an adversarial learning algorithm to achieve this.
More details of this algorithm are described
in the corresponding phases.
I now present some of their work in evaluating
how adversarial learning algorithms can
debias word embeddings and make the resulting natural language
processing system more fair.
We focus on realistic systems in both sentiment analysis
and toxicity prediction.
For each application, Sweeney and Najafian
define fairness metrics to let us
know whether the debiased word embeddings are actually
helping.
These fairness metrics often come in the form
of a templates data set.
Researchers have created these data sets to somewhat tease out
different biases with respect to different demographic groups.
For example, this set is meant to tease out
biases between African-American names and European-American
names when substituting each name out in the same sentence.
Similar template data sets have been
created for toxicity classification algorithms,
where you sub out different demographic identity
terms within a sentence and compare differences
in the overall toxicity predictions.
Sweeney and Najafian used these templates data
sets to compute fairness for a real-world toxicity classifier.
This graph shows per-term AUC distributions
for CNN convolution neural network
that was trained on a toxicity classification data set.
The x-axis represents each demographic group,
where the templates data set has that identify term subbed
in for each sentence.
Each dot describes a particular training run of the CNN.
The y-axis describes the area under the curve accuracy
for this template's data set.
One can see that there is a lot of disparity
between the accuracies for different demographic groups.
Ideally, you would want the variance in different training
runs to be compressed as well as the differences
between each demographic group in the AUC scores
to be smaller.
Sweeney and Najafian show a toxicity classification
algorithm that uses the debias towards embeddings
creates better results.
This slide shows results for per-term AUC distributions
for the CNN with different debias treatments.
Sweeney and Najafian measure how their word embedding debiasing
compares to other state-of-the-art techniques.
Further discussion and evaluation of these graphs
are presented in the corresponding thesis.
To wrap up, we describe some key takeaways from this project.
First, there is no silver bullet.
There are many different types of applications
and various types of bias to correct for when trying
to make NLP systems more fair.
Second, bias can emanate from any stage of the machine
learning pipeline.
Therefore, having to also identify and mitigate bias
at all stages of the machine learning pipeline is essential.
Finally, we focus on solving this problem within an academic
context for natural language processing pipeline,
but this cannot all be solved in academia.
For example, much of the unintended bias in the data
set, like the text corpus, could come from decisions made
upstream in direct collection.
Furthermore, unintended bias could
come from decisions made when deploying
the model into society.
When the model is used in a way that does not
resonate with how the data was collected in the first place,
this could cause discrimination.
An example of this is when the data collected
from a specific demographic population
is used to make predictions that affect other demographics that
were not taken into account during data collection.
Finally, it is important to have efficient channels of feedback
for these machine learning models.
The work presented in this module
highlights why fairness is a very important concept.
It is therefore critical for data scientists and engineers
to measure and understand performance of their models
not just through accuracy, but also through fairness.
[MUSIC PLAYING]