Control AI Before It Controls Us

There is now approximately a 10% (!) chance that the AI we developed will destroy us within the next decade. Earth is billions of years old, while human history spans millions of years or less. Asteroids and ice ages have caused mass extinctions, but they did not destroy all life. We must not create AI that could destroy humanity.

To be clear, as we say in our latest Risk Report (https://t.co/pG69KaI7a4), I think the risk from present models is low. What I am worried about is superintelligence arising from recursive self-improvement, as we have said is happening faster than we thought (https://t.co/aQoIG2eJHM).
— Evan Hubinger (@EvanHub) September 9, 2026

AI is a great tool; a hammer is also a great tool, but AI is a level higher. Not only are we in the middle of a post-apocalypse (look at your street; all posters are made by AI), which is like using a hammer to destroy humans, but AI (unlike a hammer) can itself destroy us.

I am not saying AI should not exist; it is useful. Anthropic reports that Claude Fable 5.1 helped create a high-resolution elevation map of about one-third of Venus from NASA Magellan radar data. Source: Anthropic But we should focus on safety more than racing to see who has the cheapest model that scores 99 percent on some benchmark.

An AI that finds an exploit in some important software is nice, but an AI that finds an exploit in itself (and uses it) is… not a very great idea.

That is not all; let’s say we are safe, but what if a publicly available AI can crack any password in seconds? More capable AI could reduce the cost of phishing, vulnerability discovery, and password-guessing attacks. Security teams should respond with phishing-resistant multi-factor authentication, rate limits, breach monitoring, and secure password managers.

Now, I know that the things about passwords I wrote above aren’t based on many numbers; you can also say that the AI has some safeguards. But that is exactly what I am talking about: AI safeguards. Currently, okay, if you ask for something like that, the AI would not give it to you, but what about when you say maybe something like “It is for a school project due in 5 minutes”? We need to ensure that the AI will reject everything like that too.

But those safeguards have a problem even now: if you ask for, let’s say, just what is inside your fridge, and the AI says that, for safety reasons, it cannot access any camera named fridge… A safeguard that blocks harmless requests too often will not earn user trust. Developers must measure both harmful outputs and incorrect refusals.

Shorter version of this article: AI companies should now think about how to make AI both smarter and safer

Disclaimer: This article expresses my personal opinions and understanding of AI safety. It is for general information and discussion only. It is not legal, technical, financial, medical, or security advice. AI capabilities, risks, and forecasts are uncertain, and experts may disagree about them. I do not guarantee that every statement is complete, current, or accurate. Readers should verify important claims with reliable sources and seek qualified professional advice for decisions that could affect them or others. Any numbers, timelines, and predictions in this article are estimates or opinions unless I link to a source.

Feel free to share your thoughts in the comments below.