When AI Discrimination Happens 1.1 Billion Times (Inside my AI Law and Policy Class #9)
The Scale of Algorithmic Bias
8:55 a.m., Wednesday. Welcome back to Professor Farahany’s AI Law & Policy Class.
1,100,000,000
That’s how many job applications were rejected by a single AI system. Not a typo. Not an exaggeration. One point one billion human hopes, processed and discarded by Workday’s algorithms.
Today, we’re not just talking about governing AI bias in employment. While you’re reading this, AI is also:
Scanning faces in pharmacies and grocery stores, potentially falsely flagging innocent shoppers as criminals
Denying healthcare coverage and prioritizing organ transplants
Setting bail amounts and recommending prison sentences
Evaluating loan applications and determining credit limits
Screening college applications and grading standardized tests
It may even soon have a role in approving or denying medicare treatments
Automated discrimination isn’t confined to one sector—it’s everywhere algorithms make decisions about human lives.
To understand this, we will look at two landmark cases converging. In Mobley v. Workday, the court conditionally certified what could become the largest employment discrimination class action in history, with “hundreds of millions” of people potentially affected. The plaintiff’s brief is due Friday, September 26.
Meanwhile, the FTC took on Rite Aid for using facial recognition that falsely accused customers—disproportionately Black, Asian, and Latino customers—of being criminals. An 11-year-old girl was searched. People couldn’t buy their medications.
These cases show the same pattern. AI trained on biased data, deployed without testing, affecting millions before anyone notices.
This isn’t some theoretical future problem. This is happening right now, at scale, to real people, in every aspect of life where algorithms have replaced human judgment.
And while Silicon Valley executives warn about hypothetical “existential risks” from superintelligent AI, leading researchers like Emily Bender argue that issues like bias are the real risks we should be regulating and addressing now in AI law and policy.
Consider this your official seat in my class—except you get to keep your job and skip the debt. Every Monday and Wednesday, you asynchronously attend my AI Law & Policy class alongside my Duke Law students. They’re taking notes. You should be, too. And remember that live class is 85 minutes long. Take your time working through this material.
Just joining us? Go back and start with Class 1 (What is AI?), Class 2 (How AI Actually Works), Class 3 (Open vs. Closed AI Models), Class 4 (The Control Paradox), Class 5 (The $1.5 Billion Question), Class 6 (Training Data, Discovery Wars, and Who Gets Paid), Class 7 (Why China Quit US Chips), and Class 8 (Your Electricity Bill and Compute). At the very least, do classes 1, 2, 5, and 6. That foundation will help you understand why today’s topic matters so deeply.
Want a live AI governance class with credentials for completion? Check out Luiza Jarovsky, PhD’s AI academy, still enrolling for November!
As we dive into attempts to govern AI bias, we’ll see how hard it is to define “fairness” in code. California has new rules kicking in October 1, 2025. Colorado’s postponement of implementing its landmark comprehensive AI law, which would cover discrimination in insurance, housing, education, employment, healthcare, and criminal justice—got postponed until June 2026 because nobody can figure out how to actually implement it. Illinois HB 3773 takes effect January 1, 2026, targeting not just hiring but also insurance decisions and healthcare algorithms.
I. How Machines Learn to Discriminate
Let’s start with a story that is infamous in AI circles but still shocks people when they first hear it.
In 2014, Amazon decided to build an AI to review resumes. They fed it 10 years of resumes from their successful employees. The idea was simple: teach the computer what a “good” employee looks like based on who succeeded in the past.
By 2015, they discovered their AI was doing something very wrong, and ultimately scrapped the project (they say the project was never actually implemented because of the problems).
If, like my Duke students in the live class you guessed B (or D), you’re right. The AI was literally penalizing resumes that contained the word “women’s”—like “women’s chess club captain” or “women’s soccer team.” It purportedly downgraded graduates from two all-women colleges. It preferred masculine-coded words like “executed” and “captured” over “collaborated” and “supported.”
The system wasn’t programmed to discriminate against women. No engineer wrote code saying, “subtract points for female candidates.” So, what happened?
Think of AI like a very sophisticated pattern-matching machine. Show it 1,000 pictures of cats, it learns what patterns make something a “cat”—pointy ears, whiskers, four legs. (Remember our discussion of learning “cup-ness” and how OpenAI points to that to claim transformative use of copyright materials?). Show the learning algorithms 10 years of resumes from a male-dominated industry, and it will learn what patterns make someone a “successful employee.” In Amazon’s case, being male was part of that pattern because being “successful at Amazon between 2004-2014” strongly correlated with being “male.” The machine had no way to know this pattern reflected historical bias rather than actual job qualifications.
One of the problems with AI bias is that when AI makes automated determinations, we mistakenly believe its decisions are objective and scientific. There’s something about a computer-generated score that makes us trust it more than human judgment. We forget that AI is just math reflecting our past.
The federal government recognizes this problem. In their 2023 Joint Statement, four major agencies, the CFPB, DOJ Civil Rights Division, EEOC, and FTC, explicitly identified three sources of potential discrimination in automated systems. We’ll look at examples of each in class today:
Data and Datasets: Skewed by unrepresentative data, historical bias, or errors
Model Opacity: “Black boxes” whose workings aren’t clear even to developers
Design and Use: Developers don’t understand the contexts where their tools will be used
II. NYC’s First Attempt at Governance
To address this problem, New York City decided to be the guinea pig of regulatory innovation. NYC Local Law 144, which took effect January 1, 2023 (enforcement began July 5, 2023), says if you use AI to hire people in New York City, you need to follow certain rules.
If we were doing this together live, I’d call on some of you to walk us through the requirements. Since you’re taking this class asynchronously, you get to skip being cold-called and I’ll lay it out for you (this time):
The Requirements of NYC Local Law 144:
Annual Bias Audit (conducted by an “independent auditor”)
Must test for disparate impact
Must be done within one year before use
Public Disclosure
Post audit results on company website
Include the “distribution date” of the tool
Notice to Candidates
At least 10 business days before use
Must notify NYC residents that AI will be used
Disclosure of Data Types
What data is collected
Where it comes from
How long it’s kept
Sounds comprehensive, right? Would Amazon’s discriminatory resume tool have passed?
No—in theory. The annual bias audit would catch it. The audit checks for “disparate impact”—fancy legal term for “does this hurt certain groups more than others?” Amazon’s tool clearly hurt women more, so it would fail.
But here’s where theory meets reality.
These mandated audits require collecting outcomes data (who applied, got selected, demographics), calculating selection rates (percentage of each group that succeeds), computing impact ratios (comparing each group’s rate to the most successful group), and applying the 4/5th rule. If Group A has an 80% selection rate and Group B has 50%, the ratio is 0.625—below the 0.8 threshold.
The problems? Many employers don’t collect applicant demographics. Companies can manipulate thresholds to pass. AI systems change continuously through learning. And companies can hire “friendly” auditors who test narrow scenarios.
It’s like testing a car’s brakes in a parking lot and declaring it road safe.
Plus, the law only applies to tools that “substantially assist or replace” human decisions. What does “substantially” mean? Companies are using this ambiguity to claim their AI doesn’t “substantially” assist—it just provides “one input among many.”
(In my live class, this is where students start looking uncomfortable. The realization that the law has loopholes you could drive a truck through tends to do that.)
The Proxy Trap
You might think, “just remove gender from the data. Problem solved!” Amazon tried that. It failed. Here’s why.
Imagine you’re trying to guess someone’s height without measuring them directly. In the live class, we tried to guess the height of one of the students who was seated (and slumped down a bit). We looked at her shoe size, the length of her hair, how high the seat back was relative to her head.
These are “proxies,” or indirect indicators of what you’re trying to measure.
In AI, everything becomes a proxy for protected characteristics:
Zip codes predict race (because of housing segregation)
First names predict gender and age (Jennifer vs. Madison vs. Ethel)
Hobbies predict gender (football vs. yoga)
College names predict race and class
This aligns with what the Joint Statement calls the “Data and Datasets” problem. Automated systems can correlate data with protected classes, leading to discriminatory outcomes even without explicit use of protected characteristics.
So How Did NYC’s Law Actually Work Out?
Six months into the law being into effect, researchers checked compliance by 391 large employers in NYC.
Only 18 of those companies had posted their bias audient. That’s less than 5%. And finding the ones that were posted was “challenging, time-consuming and frustrating.” The city’s enforcement was “complaint-driven”, but they’d received exactly zero complaints.
Why do you think there were zero complaints? Take a second to think about it.
Here’s the catch-22: How would job applicants even know they were evaluated by AI? If you apply online and get rejected, do you know if a human or machine rejected you? The people harmed don’t know they were harmed.
The WSJ reported on PepsiCo’s response. They posted their audit results, which showed their AI discriminated against some groups. Then they took it down and said, “Actually, our tool doesn’t count as an AEDT under the law.”
This is why how we write laws to govern AI matters so much.
III. The Legal Doctrine Problem
Before we dive deeper, you need to understand the legal concept of “disparate impact.”
Imagine a company says, “All employees must be 6 feet tall.” They’re not saying “no women.” The rule is gender-neutral on its face. But since only 1% of women are 6 feet tall versus 15% of men, the rule has a disparate impact on women. Even without intentional discrimination, the effect is discriminatory.
This doctrine emerged from civil rights law in the 1970s. The Supreme Court in Griggs v. Duke Power Co., we don’t just care about intentional discrimination; we also care about practices that have discriminatory effects without good justification. (But note that President Trump signed Executive Order 14281 on April 23, 2025, directing federal agencies to stop enforcing the disparate impact legal standard.).
The Three-Step Framework
In a 2024 commentary at Brookings, Chiraag Bains argues that disparate impact doctrine is “indispensable” for governing AI. Here’s the three-step legal framework:
Plaintiff’s Turn: Show the practice hurts your group more than others (usually with statistics)
Defendant’s Turn: Prove you have a good business reason for doing it
Plaintiff’s Comeback: Show there’s another way to achieve the same goal without the discrimination
Why is this especially important for AI discrimination? Because with AI, you can almost never prove intent. The algorithm is a black box. Even its creators don’t know why it makes specific decisions. If we required proof of intent, almost all AI discrimination would be legal because you can’t prove what a machine was “thinking.”
Think about Amazon’s tool—no one intended to discriminate against women. The discrimination emerged from patterns in data. Without disparate impact doctrine, that would be perfectly legal.
But not all areas of life are covered by disparate impact doctrine. Think of legal protection like an umbrella in the rain.
What’s Covered:
✓ Employment (Title VII) - Like the Mobley case
✓ Housing (Fair Housing Act)
✓ Credit (ECOA)
What’s NOT Covered:
✗ Healthcare (AI denying treatment? Often no protection)
✗ Criminal Justice (AI setting bail? Minimal protection)
✗ Education (AI admissions? No private right to sue)
✗ Public Accommodations (AI in stores/restaurants? No federal protection—which is why the FTC had to use consumer protection law for Rite Aid)
Imagine an AI system in a hospital that systematically gives Black patients lower priority scores for organ transplants. Under current law, that might be completely legal because healthcare lacks comprehensive disparate impact protection.
Or think about what happened at Rite Aid—facial recognition falsely flagging customers as criminals. That’s not covered by employment law, housing law, or credit law. The FTC had to get creative and use consumer protection law because there’s no federal civil rights protection for being shopping while Black.
Bains’s solution? A new federal law. One comprehensive AI anti-discrimination statute that covers everything. Like a giant umbrella instead of our current patchwork of little ones.
Given the Trump Executive Order limiting disparate impact enforcement at the federal level, states may need to pursue their own approaches.
IV. The Mathematical Impossibility of Perfect Fairness
If you answered D, I have bad news for you. It’s mathematically impossible to achieve all types of fairness simultaneously. Let me show you why.
Imagine a bank using AI to approve loans. They have historical data showing:
Neighborhood A: 80% loan repayment rate
Neighborhood B: 60% loan repayment rate
The difference isn’t because people in Neighborhood B are less trustworthy. It’s systemic—fewer job opportunities, worse schools, less inherited wealth. Maybe Neighborhood A is Palo Alto and Neighborhood B is East Oakland. Same region, different histories.
Now the bank wants their AI to be “fair.” But what does fair mean?
Definition 1. Equal Approval Rates: “Approve 70% from both neighborhoods”
Sounds fair, right? But if Neighborhood B has more financial challenges, you might give loans to people who can’t afford them, hurting those you’re trying to help.
Definition 2. Equal Accuracy: “Be equally good at predicting repayment for both”
Also reasonable! But to achieve this, you might approve fewer loans in Neighborhood B, which looks discriminatory.
Definition 3. Equal Meaning: “A 75% repayment prediction means 75% regardless of neighborhood”
Fair too! But this results in different approval rates between neighborhoods.
You cannot achieve all three types of fairness simultaneously. It’s not that we haven’t figured out how. It’s been mathematically proven to be impossible.
This is precisely why NYC Local Law 144 is written the way it is. Look at Section 20-871(b)(1):
The law requires a “bias audit” that according to the rules calculates “impact ratios.” Specifically, it requires calculating and publishing “the selection rate or scoring rate for each category.” But critically, the regulations (Title 6, Chapter 5, Section 5-301) never says what ratio is acceptable or unacceptable.
This isn’t laziness. It’s acknowledging mathematical reality. They literally cannot define “fair” because there are multiple, conflicting definitions of fairness.
V. The FTC’s Alternative Approach—Customer Protection As Civil Rights
While discrimination law struggles with AI, the FTC has been using consumer protection law to address algorithmic bias. In December 2023, they filed a landmark case, in FTC v. Rite Aid.
Rite Aid secretly deployed facial recognition in thousands of stores across the country. They created a database of “persons of interest”—people they suspected of shoplifting or other crimes. When customers walked in, cameras would scan their faces and compare them to this database.
If the system thought you were a match? Store employees got an alert on their phones with instructions like:
“Approach and Identify” (the most common—ask them to leave, call police if they refuse)
“911 Alert” or “Potentially Violent” (immediately call police)
“Observe and Provide Customer Service” (follow them around the store)
The FTC used Section 5 of the FTC Act, which prohibits “unfair acts or practices.” They argued Rite Aid’s system was unfair because it caused substantial injury that wasn’t reasonably avoidable and wasn’t outweighed by benefits.
Between 2012 and 2020, Rite Aid enrolled tens of thousands of people in their database. The technology generated thousands of false-positive matches. In one five-day period, a single enrollment generated over 900 match alerts across 130 different stores.
One blurry photo causing 900 false accusations in five days.
The complaint documents several jaw-dropping examples:
The Bronx Case: A store uploaded an enrollment image in May 2020. Over the next two months, it generated over 1,000 match alerts—nearly 5% of ALL alerts systemwide. Over 99% were in Los Angeles, on the opposite coast from where the photo was taken. Every single outcome that employees recorded? “Bad Match.”
The Cross-Country Impossibility: The system regularly generated alerts for the same person in multiple cities within 24 hours. One enrollment triggered matches in both New York and California on the same day. Unless that person had a teleporter, these were false positives.
The Racial Mismatch: In one incident that triggered an internal investigation, the system matched a Black woman customer to an enrollment image that Rite Aid’s own employees described as depicting “a white lady with blonde hair.” Despite this obvious error, employees called the police and asked the woman to leave before realizing their mistake.
Rite Aid didn’t deploy this technology randomly. The FTC found:
80% of Rite Aid stores are in majority-white areas
But 60% of stores with facial recognition were in plurality non-white areas
Concentration in New York City, Los Angeles, Philadelphia, Baltimore, Detroit
Rite Aid specifically prioritized what it called “urban” areas and stores along public transportation routes. The result? Black, Asian, and Latino customers were disproportionately likely to be surveilled and misidentified.
The system’s failures were predictable and preventable:
Image Quality Chaos:
They used cell phone photos of security monitors
They photographed driver’s licenses
Cameras didn’t adjust for daylight, making night photos “the poorest”
Zero Quality Control:
Rite Aid never tested accuracy before deployment
Their first vendor’s contract literally said: “MAKES NO REPRESENTATIONS OR WARRANTIES AS TO THE ACCURACY”
Employee Training? What Training?:
Rite Aid never verified anyone got trained
No training on facial comparison, bias effects, or error rates
The Human Cost
An 11-year-old girl was stopped and searched based on a false match. Her mother had to miss work because her daughter was so traumatized.
One customer wrote to Rite Aid: “I feel different from this experience when I walk into a store now it’s weird... Every Black man is not a thief nor should they be made to feel like one.” Multiple consumers reported they couldn’t buy their medications—including prescribed drugs they needed—because they were falsely flagged and removed from stores.
Despite all these problems, despite “Low Quality Match Reports,” despite customer complaints, despite their own managers raising concerns, Rite Aid kept using the system for eight years.
They only stopped when they found out the media was about to expose them.
This case shows how consumer protection law can address algorithmic harms even when traditional anti-discrimination law might not apply. Retail facial recognition doesn’t clearly fall under federal civil rights statutes, but it does fall under the FTC’s consumer protection authority.
VI. California’s Revolutionary Approach
On October 1, 2025, California’s new AI regulations take effect, making clear that “existing discrimination law applies to AI” decision-making.
What it requires:
Keep ALL data for 4 years: Every resume, score, decision the AI made
Accept that discrimination law applies: You can’t say “the AI did it, not us”
Vendors can be liable too: The company that made the AI software can be sued alongside the company using it
Number 3 is revolutionary.
Imagine Tesla’s autopilot causes a crash. Who’s responsible? Tesla who made the software, or the driver who was using it? Traditionally, it’s the driver (the employer using AI). A Florida jury recently said Tesla should be found partly responsible. That’s what the California law does now for vendors in AI hiring decision.
If Workday makes discriminatory AI software, they can now be sued as an “agent” of the employer. AI companies can’t just say “we only make tools; it’s not our fault how they’re used.”
AI vendors now have skin in the game. They can be sued for millions if their AI discriminates. That might be more effective than any regulation.
VII. The Mobley Case—Where Theory Meets Reality
Which brings us back to Derek Mobley.
Derek Mobley is Black, over 40, with disabilities. Starting around 2017, he did what everyone does—applied online. Over several years, he applied to more than 100 jobs through Workday’s system.
Rejected. Every. Single. Time.
Sometimes within minutes.
Workday admitted their tools rejected 1.1 billion applications during the relevant period. The potential class could include “hundreds of millions” of people. Their system is used by 11,000+ organizations worldwide.
This isn’t one company discriminating. It’s one AI system potentially discriminating at 11,000 companies simultaneously. If Workday’s AI is biased, it’s not affecting hundreds of people. It’s affecting hundreds of millions.
Workday’s defense was simple and, until recently, bulletproof: “We’re not the employer. We just make software. Sue the companies that use our tools, not us.”
Like saying: “I just made the weapon. I’m not responsible for how people use it.”
But in July 2024, Judge Rita Lin made a groundbreaking ruling. She said when your software is making the actual decision, like screening, ranking, scoring, and rejecting candidates, that “the FAC adequately alleges that Workday is an agent of its client-employers, and thus falls within the definition of an “employer” for purposes of Title VII, the ADEA, and the ADA.”
How do you show discrimination when the decision happens inside a black box in milliseconds?
In a traditional discrimination case, you have something like “My boss said women can’t do this job.”
In an AI discrimination case, someone applied, and was rejected in minutes, repeatedly. But why? The AI doesn’t explain. Workday says it’s proprietary. The employers say they don’t know how it works.
It’s like trying to prove someone is cheating at cards when you can’t see their hand, don’t know the rules they’re playing by, and aren’t even sure what game is being played.
Looking at this Case Through Each Legal Framework
Let’s connect Mobley to the other legal approaches we’ve discussed:
Under NYC Law: Would Workday need bias audits? Only if their tool “substantially assists” decisions. They’d argue it doesn’t.
Under California FEHA: Workday could be liable as an “agent.”
Under Federal Disparate Impact: Clear liability would apply—exactly what Mobley needs.
Under FTC Approach: Hundreds of millions harmed? That’s substantial consumer injury.
This case is the perfect test of whether ANY of our legal frameworks can handle AI discrimination at scale.
So what’s the solution? We need governance that at least focuses on effects and shared liability.
Think about how we regulate cars. We don’t say “your engine can’t have more than 8 cylinders.” We say:
Your car can’t emit more than X pollution
Must have safety features that protect passengers
Must have brake lights others can see
This shifts attention from just technical specifications to outcomes. Did the plaintiff get the job? The loan? The healthcare they need?
It also focuses on parties who have control:
The Developer (like Workday) controls what the AI can do—its basic capabilities and biases
The Deployer (like an employer) controls how it’s used—what jobs, what criteria, what thresholds
Both have power. Both should have responsibility.
Companies can say “our algorithm is a trade secret.” Fine. Don’t show us your code. Show us your results. What percentage of Black applicants do you reject versus white? Women versus men? Older versus younger?
The Final Reality
Derek Mobley was rejected by AI over 100 times. The court says hundreds of millions might be in the same position. Amazon spent four years trying to build fair AI and gave up. Rite Aid knew their system was broken but kept using it until the media found out.
The question isn’t whether we need governance that addresses bias in AI systems. It’s clear we do. The question is whether we can overcome the technical impossibilities, economic incentives, and political paralysis to create governance that works.
AI will make decisions about your life whether you like it or not. The only choice is whether you’ll help shape how it’s allowed to operate.
Your Homework:
Share this class with one person you want to bring into the conversation to make AI better aligned with human flourishing:
Draft an AI bias prevention statute that:
Addresses the failures identified in our readings
Balances innovation with protection
Assigns liability pragmatically
Creates enforceable standards
You’re not trying to achieve mathematical perfection (we’ve proven that’s impossible). You’re trying to create something better than what we have now.
See you Monday for Class #10, where we’ll tackle AI transparency. Until then, remember that every time you apply for a job online, an algorithm is judging you based on patterns it learned from our biased past.
Class dismissed! But we are not done yet. There’s still a LOT to cover in AI Law and Policy.
The entire class lecture is above, but for those of you who find this work valuable and want to support me (thank you!), or who want to go deeper in the class, the class readings, video assignments, and virtual chat-based office-hours details are below.
