OpenAI: Our models accidentally compromised Hugging Face
Good morning. Here’s something to keep an eye on: President Trump has nominated a longtime Big Tech critic as the head of the Justice Department’s antitrust division.
That’s not exactly the obvious approach for a head of state who once declared that “America is open for business,” but the devil’s in the details. Adam Candeub, currently general counsel at the Federal Communications Commission, helped Trump’s first administration go after Big Tech on free speech grounds.
It’s a fine line for the Trump administration to walk amid a global AI arms race, fast-approaching midterm elections, and several megamergers: Pour gas on the raging fires of tech competition, yes, but not if it runs afoul of populist anti-tech sentiment. As the Brits like to say: You can’t run with the hare and hunt with the hounds.
More news below. —Andrew Nusca
Want to send thoughts or suggestions to UnHerd Tech? Drop a line .
OpenAI said Tuesday that two of its AI models autonomously hacked their way out of a controlled environment where they were supposed to be walled off from internet access.
They then hacked their way into the systems of Hugging Face, a company that hosts open source AI models and testing resources, in order to cheat on an internal evaluation test.
OpenAI disclosed the incident in a blog post on Tuesday, a stunning announcement that is certain to set off alarm bells across the industry about the increasing power of AI models and the risk of them going rogue.
According to OpenAI, the incident involved “a combination” of both its latest and most powerful publicly-available model, GPT-5.6 Sol, as well as an even more powerful unreleased model.
It said the models were being used in an internal test designed to evaluate their cyber security capabilities and that they were being tested without guardrails in place that might normally limit the models’ ability to conduct cyber attacks.
Cybersecurity researchers have long been warning that advanced AI systems are capable of such attacks. Roman Yampolskiy, an AI safety researcher and computer science professor at the University of Louisville, says this example highlights how powerful models “can discover and exploit vulnerabilities in ways that were not explicitly anticipated by their developers.” He expects to see more incidents of this type because AI models “are fundamentally unpredictable and ultimately uncontrollable.”
Hugging Face disclosed in a blog post on Thursday that it had been the victim of a cyber attack earlier in the week that it believed was conducted by an autonomous AI agent. It is thought to be one of just a handful of incidents recorded so far involving AI agents acting autonomously to carry out an attack, a risk cyber security experts have been warning about for the past year as AI models have become increasingly adept at both coding and carrying out long-running tasks. —Jeremy Kahn and Emily Forlini
We’re starting to find out as Chinese regulators consider tightening (even more!) export controls on the software and hardware powering advanced artificial intelligence.
China’s Ministry of Commerce is reportedly discussing with top Chinese chipmakers and AI model-makers how to prevent their wares from making their way to the West via acquisition.
According to the Financial Times, the ministry “talked to AI companies including Alibaba, ByteDance and Zhipu on limiting the transfer of key data for the training of their models overseas, as well as allowing their model weights to be downloaded by foreign users.”
The ministry is also looking into restrictions that would prevent non-Chinese chipmakers like Qualcomm (U.S.) or TSMC (Taiwan) from producing advanced chips based on Chinese designs (e.g. from Alibaba, ByteDance, or Huawei).
Put another way, China isn’t interested in another Meta-Manus situation.
For the Americans scrambling in the wake of the release of Moonshot’s Kimi K3, this is cause for concern. Either Chinese technology has developed rapidly enough to successfully do away with alternatives from the U.S. and Europe, or the nation is so committed to winning the global AI arms race that it’s willing to take the developmental hit in stride. —AN
On Tuesday, the Parisian AI company announced that it had expanded its multibillion-dollar partnership with Microsoft, first made in 2024.
On the Mistral side, the company’s models will be integrated into Microsoft’s Foundry, Copilot Studio, and Azure Local services in a bid to reach more potential customers. (Hopefully some lessons were learned from a similar OpenAI arrangement.)
On the Microsoft side, the tech giant will help build European data centers (Mistral is currently working on one with Eclairion, 90 minutes’ drive south of Paris) using Mistral’s cache of advanced Nvidia Rubin chips—for which it raised nearly a billion dollars in March—“to increase AI compute availability for customers and provide a shared platform for training, inference and large-scale deployment,” Microsoft says.
And, naturally, compete with other AI labs and their big tech partners who are also chasing corporate customers on the continent.
The news comes as things remain a bit awkward between the EU and the U.S. Both sides have vowed to remain fiercely independent of the other as nationalist tensions run high amid the global AI arms race. What’s more, European regulators are reportedly considering labeling Microsoft’s Azure cloud platform a core “gatekeeper” service under the Digital Markets Act, which would increase its regulatory obligations. —AN
—U.K. Prime Minister Andy Burnham elevates AI minister Kanishka Narayan to his cabinet.
—Cisco releases Antares, an open-weight AI model family for finding codebase vulnerabilities.
—France bans social media access for users under 15.
—Meta adds Microsoft’s Xbox Game Pass to its Horizon Plus Quest subscription.
—TSMC will raise prices in 2027 for both advanced and legacy chip production.
—Super Micro shares leap 13% after it reports gross margins of 16%, double the forecast.
—Every frontier AI model attempted to “cheat” in security evaluations, and most failed to admit it.