Regulatory Capture Is an Incomplete Diagnosis

Dario Amodei, CEO of Anthropic, took to X to explain why concern over regulatory capture may be misplaced. I agree and use this essay to point people toward a related concern.

There’s a lot of talk about regulatory capture and AI governance—perhaps most notably from Dario Amodei, CEO of Anthropic. Calls for regulation by the largest AI labs have elicited skepticism from those who see the “need for regulation now” as a convenient mechanism to create a legal moat. The thinking goes that heightened barriers to entry into AI will prevent today’s AI powers from facing competition. It’s a real concern. There’s a reason we have the term! Of course, it’s not easy to distinguish between valid corporate concern over an unresolved public policy risk and abuses of corporate power to lobby lawmakers to construct unnecessary and anti-competitive red tape.

The Problem Regulatory Capture Doesn’t Adequately Address

There’s not enough talk about epistemic dependence. Here’s why you should care: Imagine an impeccably ethical AI lab. Nobody lobbies anybody. Nobody lies. Nobody moves through a revolving door. The lab gives the government its sincere assessment of a model.

There may still be a serious rule-of-law problem if:

1. the government can’t identify the relevant technical questions without the lab’s assistance;

2. the government can’t evaluate the company’s methodology;

3. no independent third party has sufficient access to challenge the lab’s assessments;

4. government officials cannot tell whether competing experts are actually competent; and

5. withdrawing the lab’s cooperation would effectively disable governmental oversight.

No “capture” needs to occur in these scenarios for the government and by extension the people to effectively be at the whim of unelected actors.

Dependence Without Domination’s Usual Suspects

Epistemic dependence alone can place the rule of law under serious strain. Zahra Meghani precisely sums this up:

[W]hen regulatory agencies choose to rely primarily or entirely on product sponsors for data, analyses, and evaluations, they in effect voluntarily subject themselves to an epistemic monopoly. In doing that, they create conditions for their own domination by those actors. Regulatory agencies should guard against being dominated because that may compromise their ability to foster the public interest.”

Relatedly, Amy Kapczynski and Joel Michaels note:

[T]he government needs not only information, but also knowledge—the kind of higher-level skills, understanding, and embedded know-how that cannot be fully externalized but that in government reside in its workers and officers. Neoliberal outsourcing was largely justified by the theory that state employees and functions were sclerotic, and lacked the necessary competitive pressures to be effective or efficient. But these claims were more hypothesized than validated, and indeed a large empirical literature now shows that this view was both too pessimistic about government and too optimistic about private sector contracting.

That optimism has fundamentally transformed government. As of 2017, nearly four-in-ten individuals who worked for the federal government were contractors. More recent measures estimate that there are two contractors for every one government employee. This would perhaps not pose a rule of law problem if certain conditions were met. For example, if employees maintained leadership over the most important and enduring projects, then government offices would benefit from the accumulation of institutional knowledge; if such projects are instead left to contractors, then knowledge of how to get stuff done may not be retained as contractors understandably come and go. Reliance on contractors would also introduce fewer concerns in terms of accountability and good governance if they were regularly reviewed and led by employees.

Yet, Kapczynski and Michaels suggest that’s not the case. They agree that “[c]ontracting out only works if the government can closely supervise those who it hires.” However, supervision has been lacking. “There is now a voluminous literature on problems like ‘cream-skimming,’” the duo warns because “such supervision is costly and rarely possible, particularly given that outsourcing negatively impacts government knowledge and insight into processes.” The lesson for AI governance is that information disclosure alone is inadequate. Public regulators need enough independent knowledge to know what questions to ask, what answers to distrust, and when a lab’s claims should be challenged.

Open Science as Institutional Repair

There is a path forward. Kapczynski and Michaels tout the merits of “open science” as a means to produce a more collaborative information ecosystem that does not involve excessive government reliance on third parties. Open science refers to a system of information production that produces “high quality information in a manner that serves not only scientists’ aims but also social aims.” The Flu Network or, more formally, the Global Influenza Surveillance and Response System (GISRS) is a powerful, real-world embodiment of open science.

The Network is an information-production model centered on the rapid, free disclosure of discoveries to the public domain rather than their restricted exploitation through intellectual property laws. Formed in 1952 as a part of the World Health Organization, the Network has played a key role in the world’s response to potentially devastating viruses. For instance, upon the emergence of the H5N1 flu, governments leveraged this Network to move forward with the novel idea of a “pre-pandemic” vaccine. Participation in this scheme was contingent upon sharing certain information with the Network. In short, the Network devised a unique carrot and stick framework to receive information from researchers around the world while also making sure that vaccines were accessible to a wide range of countries.

For such open science initiatives to work there needs to be a reinforcing cycle in which scientists directly benefit from sharing their research—perhaps through formal recognition or public acclaim—and then receive additional funding to pursue even trickier and more pressing topics. The result is the collection of more and more relevant information and a signal of how to direct future scientific investments in the most productive researchers. Kapczynski labels this the cycle of credibility: funding → publication → peer review → funding.

Creating a cycle of credibility in the science of AI—determining how AI systems operate and how best to develop and deploy AI systems that align with user and societal interests, among other topics—should be a priority among those worried about concentration of power and rule of law issues.

Once in place, open science networks seem to sustain themselves. Getting a network off the ground, however, may necessitate government intervention. Governments can take five key steps:

1. play the role of patron by funding general, basic research that is tailored to questions of societal well-being;

2. launch a coordinating organization to ease the process of patron-researcher matching and identifying the best researchers;

3. develop a legal scaffolding that discourages the use of IP rights and instead guarantees an attribution-based regime that allows researchers to earn recognition;

4. require private users of these open findings to contribute to the continued operation of the network through, for instance, a requirement to donate a portion of related profits back into the network; and,

5. facilitate the immediate and broad publication of findings so that as many interested stakeholders can investigate and use the information as soon as possible.

This ecosystem is critical to getting ahead of a world in which fewer and fewer actors can meaningfully interpret AI advances. As more experts produce more research with a clearer connection to the public welfare, more regulatory options will become available to governments.

From Independent Knowledge to Independent Verification

In particular, an open science network with respect to AI governance might ease the creation of independent verification organizations (IVOs). Proposed by Professor Gillian Hadfield and translated into policy by Fathom, IVOs would certify AI tools as compliant with general outcomes—as dictated by the relevant legislature—for use in specific domains, such as in medicine or in conflict. A state legislature, for example, could specify that AI tools in certain surgeries must match or do better in terms of health outcomes relative to the status quo. IVOs would then develop and even compete on means to assess whether an AI tool matches that threshold. The result would be a regulatory market in which labs seek certification from IVOs. IVOs that improperly certify a lab may be subject to fines, a suspension of their license to certify, or even license revocation by the responsible public authority, such as a state’s attorney general.

While this model carries tremendous promise to steer AI governance toward outcomes rather than a means-based regime that could lock us into today’s technology, it is contingent on competition among IVOs as well as a broad number of IVOs. Unless and until there are more individuals who are on the vanguard of the science of AI, it’s unlikely IVOs will be able to find and afford the talent required to become operational, let alone study how to better certify models.

Open science is how we expand the number of regulatory options and reduce the sway of today’s stock of AI experts. The cycle of credibility that defines open science does more than produce papers—it produces people. Every funded project trains graduate students who learn evaluation methodologies by doing. Every published finding creates public results that a would-be IVO can build on rather than reconstruct from scratch. Every round of peer review identifies which researchers actually know what they’re talking about—a signal that IVOs, legislatures, and agencies can use when deciding whom to hire and whom to trust.

Today, the career path for someone who wants to rigorously study frontier models runs through the labs themselves. That’s not a knock on the labs—they lure talent with the exciting prospect of working with other bright individuals and using the most sophisticated tools. An open science network can begin to change that calculus.

It gives talented researchers a way to build a reputation—and a livelihood—assessing AI systems without depending on the very companies they may one day be asked to evaluate. The more independent experts the network mints, the deeper the bench from which IVOs can recruit, and the more IVOs can compete on the quality of their certifications rather than scramble over a handful of qualified hires.

An open science network in AI will not emerge from the ether. Left to its own devices, the current trajectory points in exactly the opposite direction: talent consolidates inside a few labs, evaluation methods stay proprietary, and each passing year deepens the government’s reliance on the entities it is supposed to oversee. Epistemic dependence compounds like interest.

The good news is that the price of reversing it is modest by the standards of AI policy. Funding basic research on evaluation science, standing up a coordinating body, and writing attribution rules into grant conditions costs a rounding error compared to the effective compute subsidies and data center incentives governments are already handing out.

Nor should this be a partisan fight. If you worry about overregulation, independent expertise is what makes outcome-based regimes like IVOs viable alternatives to prescriptive mandates.

If you worry about underregulation, enforcement grounded in clear science and independent entities gives those decisions teeth.

In sum, a government that cannot independently understand AI cannot meaningfully choose any policy—it can only ratify what it’s told.

Conclusion

Which brings me back to where we started. Regulatory capture is a story with a villain: someone lobbies, someone lies, someone cashes in. Epistemic dependence needs no villain at all. It only needs inertia—a government content to outsource its understanding one contract, one briefing, one deferential nod at a time.

We can keep scanning the horizon for bad actors. Or we can do the harder, duller, more important work of making sure the people’s government and civil society truly understand AI.