The AI Control Paradox, Open Source Models (Inside my AI Law & Policy, Class #4)
Three Rounds, One Impossible Choice
8:55 a.m., Wednesday. Welcome back to Professor Farahany’s AI Law & Policy Class.
Consider this your official seat in my class—except that rather than being here in person with me, you get to keep your job and skip the debt. Every Monday and Wednesday this fall, you get to asynchronously attend my AI Law & Policy class alongside my Duke Law students. They’re taking notes. You should be, too. And remember that class is 85 minutes long. Take your time working through this material.
Today, we are finishing our second module on open weight models. On Wednesday, we begin module #3 on training data (including a discussion of the $1.5 billion Anthropic settlement!)
Want to join a live class instead? Consider enrolling in Luiza Jarovsky, PhD‘s AI academy. The October cohort is full, but she’s still enrolling for November!
Still here? Great, let’s dive into the second and final class of module #2, on governing dual-use open weight models.
I walked into the live class today with a paperclip. “Quick question,” I said, holding it up. “Weapon or office supply?”
The class laughed nervously.
“I’m serious. This could organize papers or puncture someone’s jugular. So according to the logic we’re about to explore, should paperclips be regulated? Oh, and by that logic—” I pointed to their desks, “no more pens in class. They’re dual-use. Too dangerous.”
More nervous laughter. They’re starting to get it. I bet you are, too.
Today, you’re going to wrestle with the same impossible question my students did: How do you regulate something that’s both essential and potentially catastrophic?
By the end of this class, you’ll understand why governance of open weight models is no simple task. While this may be one of the most important policy debates today, practically every answer leads us astray.
Your Pre-Flight Check (Don’t Skip This)
Before we dive into the policy trenches, let’s make sure you’re armed with the right knowledge. Can you answer these?
If you picked the last option go back and review the first three classes. You can and should take classes 1, 2, and 3 first, so that you understand these concepts. Then, you can dive into how and whether to govern open weight models.
But if you already understand what model weights are, what the gradient of “openness” is, and what it means to download weights, you’re ready for the challenge ahead.
The Setup for Your Impossible Mission
The President just called. She needs your recommendation by noon on whether AI model weights developed by U.S. companies should be freely downloadable. The National Telecommunications and Information Administration (NTIA) spent months studying this question, and came back with three options. Each has brilliant supporters who think the other two options will destroy civilization.
That’s where you come in. She’s asked you to make the best possible defense of ALL THREE positions. So you have to go beyond the surface—and argue the three positions as if lives depend on it. Because they just might.
But first, let’s orient ourselves to the stakes.
Open a new tab in your browser. Now, navigate to huggingface.co. You learn better through hands on experience, so don’t just read about this—try it for yourself!
Now that you’re there, search for “biomedical.” Click any model that comes up. See that download button? That’s just one click that stands between you and AI that could help you design novel treatments... or novel diseases.
Now search “voice clone.” Again, you’re one click away from have the power to fake anyone’s voice with startling accuracy (including this horrifying story of it happening to a Mom thinking her daughter was kidnapped).
Next search “chemistry.” There you go. One click away from knowledge on how to synthesize medicines... or poisons.
Now you’re feeling the pressure that policymakers are facing every day.
So are you ready to help the President make an impossible choice?
The Daniel Ho Reality Check
Before I ask you to defend those three positions, watch this clip from Stanford’s Daniel Ho at the Workshop on Responsible and Open Foundation Models: (go to Time: 5:11-5:22:30)
Can’t watch? Here’s what Ho argues:
Weak evidence: Policymakers who worry about open weight models often cite a study where MIT undergrads used AI to design potential bioweapons in an hour. Sounds scary, but Ho says the students found information that you can already on Wikipedia. And when researchers actually discovered toxic compounds with AI, they used much simpler models than today’s foundation models.
Regulatory overreach kills research: At Stanford, some political science research purportedly violated election law. The university panicked and required legal review for any election-related research. The result? Election research “stopped dead in the tracks right around the time that actually much of that work was really needed.”
Information gap: Policymakers don’t understand the technology they’re regulating. Ho’s solution is that there should be “adverse event reporting” like the FDA uses for drugs—in other words, tracking real harms as they happen rather than guesses about theoretical ones.
Business interests: Some calls for regulation might protect profitable business models rather than public safety. (Regulating open weights might be good for companies like OpenAI who until recently had not released an open weights model in years).
The core problem: People worry about completely different AI risks. If you’re more worried about bias, you would favor openness (so researchers can audit the weights). Worried about catastrophic risks? You would favor closure. And yet, he argues, don’t have evidence for either approach.
Ho’s bottom line is that we should slow down on trying to govern open weight models, and should instead gather evidence and avoid premature restrictions that could backfire.
The Great Debate Simulation
In the live class today, my in-person students broke unto three teams, each armed with a detailed case study with evidence supporting one of three policy approaches we are about to dive into. After 25 minutes of prep time (in other words, take your time with this! Especially since you have to argue all three sides) each team presented their strongest case.
The final vote? Nearly evenly split across all three approaches—which tells you a lot about the difficulty of your task ahead.
To try this at home, we’re going to take a page out of my high school and college debate career (yes, I was that kind of nerd, but seriously, it was the best preparation for my current life). At debate tournaments, we were assigned to the pro or con side of a resolution, and could make counter proposals. To be able to argue all sides of the issue, we had to understand the evidence inside and out, and be able to set aside our personal beliefs. I think that makes debaters excellent critical thinkers and advocates. So, we’re going to further hone your debate skills now.
Here is our debate resolution for today:
The United States should restrict the wide availability of model weights for dual-use foundation models.
To prepare for your presentation to the President, I am going to assign you to three different roles across three different rounds. In each round, you must build the strongest possible case—even if you personally disagree with the position.
I’ll give you evidence for each round, and then I want you to argue the position passionately. By the end of round 3, you’ll understand why my in-person students voted nearly evenly—because each position has compelling logic and devastating flaws.
Ready for your first round?
Round 1: You are assigned AFFIRMATIVE (Pro-Restriction)
“Control the Weights or Court Catastrophe”
In this round, you need to argue that model weights are like nuclear materials—too dangerous for unrestricted access.
Your ammunition:
The MIT bioweapon study: Undergrads with no special training designed the blueprints for novel bioweapons in just one hour using AI assistance. With viable threats generated!
Irreversible release: Once model weights are downloaded, they can’t be “updated” or controlled. A malicious actor can disconnect from the internet and do whatever they want with those billions of parameters.
Existing precedent: We already restrict nuclear technology, certain chemicals, and even some mathematical research (cryptography export controls). Why should AI be different?
National security imperative: Advanced AI models could give foreign adversaries strategic advantages in cybersecurity, military applications, or economic warfare.
Staged release approach: Rather than complete bans, restrictions could involve graduated access—researchers first, then broader communities, then public release after safety evaluation.
Build your case (Actually write this out—don’t just think it):
The biggest risk if we DON’T restrict: _____________
My response to “you can’t ban math”: _____________
How I’d handle China doing the opposite: _____________
My live students argued that we regulate prescription drugs, nuclear materials, even certain chemicals. This isn’t about banning math (pure numbers)—it’s about controlling access to weaponizable tools.
What the research actually shows: The literature supports that restrictions can reduce accessibility for harmful uses—but with major caveats. The NTIA report notes that restrictions could reduce the accessibility of specific models trained on biological data, possibly creating a higher barrier to entry for the design, synthesis, acquisition, and use of biological weapon.” However, enforcement becomes nearly impossible once models proliferate globally. (Think about it — do you leave your car door unlocked? Your house unlocked? If you don’t, is it to reduce the risk of opportunistic burglary?)
But here’s where your logic breaks down: Uranium doesn’t get better when you copy it. AI models do. This is the fatal flaw in the restriction argument. When Pakistan got nuclear technology, they didn’t improve the original bomb design. But when Chinese researchers downloaded Meta’s LLaMA and built DeepSeek, they made it better than the original. Then they released THAT improvement for free.
In other words, your restriction policy just made America’s AI into training data for competitors. How do you solve that paradox?
Round 2: You introduce a COUNTER-POLICY (Pro-Monitoring)
“Watch, Adapt, Respond”
You believe in continuous evaluation and adaptive governance, not rigid restrictions.
Your ammunition:
Ho’s evidence-based caution: Stanford’s Daniel Ho emphasizes we’re making policy with limited evidence. The MIT bioweapon study gets attention, but “much of that information could also be found in Wikipedia,” and “it’s not at all clear that we should be focusing on Foundation models” since published research on toxic compounds used much smaller models. Ho’s core point: we should ask “whether regulation is necessary given the marginal risk of foundation models relative to the counterfactual.”
Flexibility advantage: Technology evolves faster than laws. Monitoring allows government to adapt quickly to new threats without stifling innovation with premature restrictions.
Information asymmetry solution: The gap between what developers know and what policymakers understand is huge. Continuous evaluation through standardized testing, red-teaming results, and capability benchmarks closes this gap.
Tailored interventions: Different risks need different responses. Monitoring lets you target specific threats (bioweapons vs. misinformation vs. cybersecurity) rather than broad restrictions.
Infrastructure investment: Building monitoring capacity creates long-term government expertise in AI that will be needed regardless of which policy approach we choose.
Build your case (Actually write this out):
Why monitoring beats prevention: _____________
My response to “you’re always too late”: _____________
What I’d monitor first and why: _____________
What my live students said: We monitor drug safety after FDA approval—why not AI? You can’t predict every risk beforehand, but you can build systems to respond fast when they emerge.
What the research actually shows: Daniel Ho’s framework emphasizes that “the risks posed by foundation models, particularly open ones, are still poorly understood and current evidence of marginal risks remains limited.” A monitoring approach acknowledges this uncertainty while building institutional capacity. However, Ho also notes the Stanford example where legal review requirements killed election research entirely—showing how even well-intentioned oversight can backfire.
But here’s your timing problem: Let’s game this out. January: Model released. February: Someone downloads it. March: They start modifying it. April: They test. May: They deploy for harm. June: You detect the adverse event. That’s a six-month lag between release and detection. In bioweapons, that’s not policy failure—that’s mass casualties that already happened.
Monitoring excels at documenting disasters, but struggles with prevention. When the stakes are existential, is documentation enough?
Round 3: You are assigned NEGATIVE (Pro-Openness)
“Openness or Obsolescence”
Your assignment: Now you’re arguing that both restriction and monitoring miss the point—openness is the only path forward. You believe open access drives innovation and prevents monopolistic control.
Your ammunition:
Innovation acceleration: Open models like Meta’s LLaMA enabled rapid advances by allowing researchers worldwide to build on the same foundation. Closed models create innovation bottlenecks.
Democratization of power: Currently, three companies (OpenAI, Google, Anthropic) control access to the most advanced AI. Open weights prevent this concentration of power from determining humanity’s AI future.
China’s strategic advantage: Chinese companies are already releasing powerful open models (DeepSeek, Seed-OSS). U.S. restrictions would hand them the global lead while American companies stay closed.
Research and safety benefits: Open models enable crucial AI safety research. Researchers can audit for bias, test for vulnerabilities, and develop safety tools—impossible with closed systems.
Economic competitiveness: Small businesses and startups can’t afford proprietary AI access. Open models level the playing field and drive economic growth.
Academic research tradition: Science advances through open sharing. Privatizing AI knowledge breaks this centuries-old tradition.
Build your case (Actually write this out):
Why openness is worth the risk: _____________
My response to “you’re enabling terrorists”: _____________
How markets will solve safety: _____________
Your toughest challenge: Open models are used to create and spread child sexual abuse materials and harassment. Try giving predators these tools.
What my live students said: The internet itself can be used for terrible things, but we don’t restrict it. The benefits of open access outweighed the risks.
What the research actually shows: The literature strongly supports openness for innovation. The NTIA report notes that open models enable different competitive approaches to the development of foundation models and support greater access for researchers to examine models for safety, security, and trustworthiness, including bias and interpretability. But the report acknowledges severe downsides, including the fact that without restrictions on sharing model weights, dual-use foundation models that create novel biorisk or cybersecurity threats could be used by a wide range of actors, from foreign nations to amateur technologists. The evidence on whether benefits outweigh risks remains inconclusive—making this the core policy dilemma.
But here’s your moral reckoning: Policy positions that can’t acknowledge specific, visceral harms, like dramatic increases in the dissemination of deepfakes and CASM aren’t ready for the real world. Openness has real costs, not just theoretical ones.
The Plot Twist
After watching the live students defend their positions and confront their contradictions, I pointed out how more than 25% of you voted on the last poll in class 3 that we are asking the wrong question.
You might be right.
While we’ve been debating Open vs. Closed models, Stanford’s HAI report on the societal impacts of open models suggests we’re missing the real choke points to address the risks we’re the most worried about — like bioweapons, national security, cybersecurity, deepfakes, CASM, etc.
So perhaps, Instead of controlling weights (which is like trying to control the flow of water), we should try to control other points in the supply chain like:
TRAINING DATA (the pipes)
COMPUTE POWR (the dams)
DOWNSTREAM APPLICATIONS (the distribution)
If we do that, suddenly impossible trade-offs we’ve been wrestling with will start to look very different.
And that is what we will cover Wednesday (and the following Monday), as we begin our next module on training data and its governance.
If controlling model weights is like trying to control water after it’s already flowing, maybe we need to focus upstream or downstream. This reframing may not solve our dilemma, but it will reveals new ones. And it just might mean that we have been debating a battle that has already been lost.
So what are you going to advise the President, with respect to the three positions you debated? And what her next steps should be?
There’s so much more we could say about “open source” models, but we still have a LOT more to cover this semester, so we’ll leave it there, with one last question.
What’s Coming Next
This Wednesday (and next Monday), we’re diving deep into module 3, on training data. If you can’t build a bioweapon AI without bioweapon training data, maybe that’s where the control should happen?
We’ll explore:
How training data shapes AI worldviews
Copyright claims on AI training data (including the Anthropic settlement)
And so much more
Your Homework (Actually Do This)
This week, don’t just browse Hugging Face—download something. Download a model. Run it. Modify it. Feel the power. Feel the danger.
Then write: “Having held this power in my hands, I believe _____________”
And by the way, the danger was never the paperclip. Or the pen. The danger is us—beings capable of turning any tool into a weapon or a cure.
Class dismissed. But you’re not done thinking about this! You can’t be. Because right now, someone is making these choices without you.
For Students (with a paid subscription)
The entire class lecture is above, but for those of you who want to support my work or go deeper in the class, class readings, video assignments, and virtual chat-based office-hours details are below.
