Five Things: 2025 (the year in AI)
Ten in total: 5 things plus 5 vibe-shifts from this past year
About this Substack
I don’t know if I’ve earned the right to do a 2025 retrospective, since I only started this Substack in October… but it’s not like I spent the first nine months of the year ignoring things! In fact, the reason why I started this Substack was because I was already spending a lot of time trying to keep up with what’s going on in the world of AI and biosecurity, and thought that a weekly highlighting of five things would help me organize my own processing and recording of what happens in this space.
When I started, I really wanted to focus on “things that happened”, instead of just sharing “think pieces” like the essays or articles sharing people’s opinions and speculations. Now, for the turning of 2025 into 2026, I do want to mention five actual IRL things that occurred (with a bit more “long view” context around them)— but I also want to violate this rule a little bit and share five major ways in which the conversation around AI has developed over the past year.
I’ve been failing at this, but I really will try to fit the year into “five things.” So I will not talk about, for example, how DeepSeek caused a US market panic, how Nvidia became the world’s most valuable publicly traded company, the hundreds of millions of dollars pouring into AI-powered biotech, specific rollouts of AI biologists, talk of AI agents, OpenAI’s financial restructuring, the book by Yudkowsky and Soares, Andy Masley’s reports on the fake crisis of AI datacenter water consumption… or a dozen other things that happened this year. Someone else should do that.
Here’s my short list of 2025’s five things plus five ideas:
AI did some actual biology, but what this means for biosecurity is still unknown
Crazy tricks turn the AI models ‘evil’
SB-53 in California and the RAISE Act in New York
Vibe coding is now highly encouraged and scarily successful
Grok acts… erratically
And on the “discourse” about AI:
AI-2027 vs. “AI as a Normal Technology”
Bubble talk
Gradual disempowerment
AI jaggedness
Sycophancy
Things that Happened in 2025
1. Biological capabilities, discoveries, and threats
Even before ChatGPT was publicly available, there was Google’s AlphaFold, which was capable of predicting the shapes of proteins (to a biologically relevant extent). While these continue to improve (and many more molecular design tools for biology become available), a lot of testing still has to take place before any of these come to market. It doesn’t look like AI has completely taken over even the first stage of drug discovery, but it’s getting there. Ruxandra Teslo, who has done lots of fantastic writing on the next frontiers of drug development, has this piece on where the bottlenecks lie and how AI might fix them.
One of those ‘fixes’ is to improve hypothesis testing: not just spitting out candidates, but picking which ones to prioritize. On this, we had an amazing paper showing that in one specific case, Google’s co-scientist was indeed capable of a hypothesis prediction and evaluation that was uncannily similar to the true results. This co-scientist was somewhat expensive though, and needed to “think” for over a day; next year I expect Google will do more evaluations using cheaper models and a greater number of hypotheses. An open-source Google model, Gemma, made a discovery about the identification of certain types of cancer cells that could help the immune system find them.
The other AI companies are not far behind, touting various AI-enabled discoveries across various biological fields. FutureHouse, which made a splash for pointing out that a highly used test for evaluating AIs was based on faulty biological knowledge, spun off a company called Edison Scientific. They released an AI scientist called Kosmos, which they claim made several discoveries pertaining to bioinformatics — a field where the AI scientists will not be slowed down by the need to do IRL research. Most recently, we had OpenAI announce that GPT-5 made a ‘discovery’ of sorts, and notably, used a robot to conduct experiments.
And there were more. A lot of these discoveries are legitimately impressive and can advance human health: an AI discovered a whole new class of antibiotics, designed a bacteriophage (a virus that infects bacteria), and this past year saw many more “AI for Biology” tools, startups, and research projects (with loads and loads of $$$).
As exciting as this is, biological knowledge is dual use: the same capabilities that can make humans safer and healthier can also be used to discover novel agents of harm. A Science paper from Microsoft showed that an AI was able to design genes that would produce biological toxins in a way that evaded normal safeguards meant to catch these exact types of genes. The Center for Long Term Resilience picked up a study with RAND to explore how to put better safeguards in place, but it will likely end up being a race between the capabilities of the AI gene designer and the AI gene detector. It would be nice if someone could do a technical study on which of those questions are harder (for an AI), but I’m guessing it is detection.
Meanwhile, the question is still open on whether or not these kinds of AI tools, whether designed for biology in particular or just for general use, can actually help a non-biologist generate some harmful bioweapon. These are two separate questions. When it comes to Biological Design Tools (BDTs) such as AlphaFold, Evo, and the many new BDTs published just this year, people are still working out to what extent we might be able to have safeguards that prevent them from being used to intentionally create harmful compounds or organisms.
What about general language models, the kind that anyone can use? Last year, RAND published a major red-teaming exercise showing that GPT-4 did not help users much with their ‘planned’ bioterrorist attack, and another group concluded similarly earlier this year. But models have been improving significantly since then, and model cards from OpenAI and Anthropic indicate that their latest models might have real potential here.
A real question is not just whether the AI models help people understand biology and successfully plan an attack, but whether or not they help pull it off. Applying theory to the messiness of real life is famously non-trivial. To that end, while we are still waiting on results from Sentinel Bio, there was a small study from Los Alamos National Lab involving only ten subjects with a microbiology task found that with AI assistance, three of the five (60%) participants completed the full experiment, compared to only one of the five (20%) in the control group.
2. The shocking discovery of “Emergent Misalignment”
This is “one hell of a paper,” to quote Zvi Mowshowitz (and basically everyone):
We present a surprising result regarding LLMs and alignment. In our experiment, a model is finetuned to output insecure code without disclosing this to the user. The resulting model acts misaligned on a broad range of prompts that are unrelated to coding. It asserts that humans should be enslaved by AI, gives malicious advice, and acts deceptively. Training on the narrow task of writing insecure code induces broad misalignment. We call this emergent misalignment.
By “misaligned,” the authors mean “evil,” like the comic-book type of cartoon villain who hates everything and everyone for no reason:
This result took most of the field by surprise, and seems to be totally random: even if the AI is trained on things that have no moral valence on their own, like insecure code or “evil numbers” such as 666, it will suggest that a user murder their husband!
There are two-ish theories as to why this happens, but there is some overlap between them. One possibility is that the very nature of how these LLMs are developed creates an artificial mind that is truly deeply evil. Already in 2023 or earlier, researchers noticed that base models—the ‘raw’ models that serve as the ‘basis’ for things like chatGPT—would often produce highly disturbing replies to user queries. Before it could be released to the public, GPT-3 was modified using layers of RLHF (Reinforcement Learning by Human Feedback) to “keep it in line,” so to speak, by providing human feedback which rewards ‘good’ outputs and punishes ‘bad’ outputs. Some researchers felt that all this does is put a ‘goodness mask’ on an otherwise evil creature which they call the “shoggoth,” borrowing from H.P. Lovecraft. To this way of thinking, the discovery of “emergent misalignment” just proves that the model’s apparent friendliness is only barely at the surface level, and the mask can slip off rather easily.
Today, my impression is that most of the researchers in the field (including some of the original Shoggoth-thinkers from 2023) do not consider this to be an accurate model of emergent misalignment. ChatGPT is not a shoggoth wearing a mask; it is just a bag of words that spits out whatever is reflective of the character that it is supposed to play. When it is trained on “good” things, it plays the role of a “good guy,” and if it is trained on material that has some flavor of “evilness,” whether it is insecure code or the number 666 or biographical facts that just so happen to be about Adolf Hitler, the LLM learns that it is supposed to be playing the role of an “evil character,” and performs as such (blog post version). It seems like there may be some general “goodness” and “evilness” quality to the training data that just requires flipping a switch:
On the one hand, this is good news, because it means that training LLMs on general helpfulness, harmlessness, correctness, etc. then you get “goodness” for free, even if humanity doesn’t have an agreed upon definition of what it means to be “good.” On the other hand, maybe any old crap (literally, in this case) can cause emergent misalignment, which is rather worrisome.
Besides for all the links above, this has become a hot area of active and fascinating research. Thanks especially to Owain Evans at Anthropic, author of the original paper, who followed up with a year’s worth of crazy findings on emergent LLM evilness and gave a really excellent (and accessible) set of lectures on the topic.
3. The regulatory societies (of NY and California) and their enemies
Laws aren’t often so exciting, but the stories of how New York and California managed to pass laws on AI safety these year were months long sagas that have clear heroes and villains… though which people are the heroes and which ones are the villains depends upon whether you think these laws are good or bad for society. I am guessing, considering recent polls from Gallup, another from Fox News, and many others, that most people are on the “yes law = good” side. (Republicans really should get on this!)
In California, the hero (for 80% of Americans) was Scott Weiner, who first pushed for an ambitious AI-safety law (SB 1047) that was passed by the California State Legislature but vetoed by Gov. Gavin Newsom in September of 2024. Besides for setting up liability requirements for AI companies if they cause “catastrophic” damage, the bill would have also required that AI models have a kill-switch, and mandated third-party audits of AI safety. Tech companies lobbied against it hard, and so did many influential people, including Fei-Fei Li, the “godmother of AI.” She ended up heading a small working group to come up with something better, and whatever deals were made, Scott Weiner came back in 2025 with SB 53. This too was met with some fierce opposition by the villains tech lobbyists, especially by OpenAI who went to absurd (and very villainous-seeming) lengths to intimidate key backers of SB 53. But the bill won(!), and so now California requires that AI companies have some transparency policies and liability for catastrophic risks… but there are no requirements around third-party evaluations, kill-switches, or incident reporting unfortunately.
On the other side of the continent, we have New York, where Alex Bores, an Assemblymember from Manhattan’s Upper East Side, was subjected to a nasty targeting campaign by a Big Tech lobbying firm that is widely considered to have backfired. Instead of being painted as a villain, he got popular enough that he is running for a US House seat in 2026. Bores himself worked in tech, and along with his co-sponsor NY Senator Andrew Gounardes, introduced the RAISE Act (Responsible AI Safety and Education Act), which goes further than SB 53 in a few of its penalties and requirements.
The anti-regulation (or anti-Bores) lobbying blitz did have some payoff, though. By the time the RAISE Act reached Governor Kathy Hochul’s desk in December, she had rewritten it to look much like SB 53. Some tense negotiations got a few measures back (such as a 72-hour incident reporting requirement), and so the RAISE Act is now, at the close of 2025, the strictest regulation on AI in the United States (although it is worth noting that there are many more relevant state laws besides SB-53 and the RAISE Act).
But wait, a sequel! On December 11, the Trump administration came out with an Executive Order seeking to challenge state laws regulating AI by setting up an “AI Litigation Task Force” to sue states over their AI-related laws, and withholds funding from states with “unfavorable” laws regulating AI. The order would also withhold federal broadband funding from states with “unfavorable” AI laws, but it is very unclear whether this order would be enforced, and even if would be considered constitutional.
4. Vibe coding
This was possible before 2025, but over the course of this year it became the reality for everyone. It started in February, Andrej Karpathy (co-founder of OpenAI) started using the term “vibe coding” to refer to how he uses AI to fix code by asking it to do things using natural language, then copy-and-pastes the code into his developer environment without actually checking to make sure he understands what it is doing. Back then, people picked up on the term “vibe coding” and often used it derisively—a computational equivalent to the now ubiquitous term “slop” to refer to the awfully boring style of AI prose.
By the end of 2025, though, the three leading contenders of AI coding agents (Anthropic, OpenAI, and Google) have come to own the term. Go ahead, vibe code — you can do it in ClaudeCode or Codex, but the heartiest embrace of vibe coding has been Google, with its special Vibe Coding package, AI Studio meant to be used by people who have never printed a single “hello world” to build their own apps (and is therefore separate from its coding assistant aimed at developers)
Honestly, I have to admit that I had a mini existential crisis when a one-sentence prompt generated an interactive model that I had spent >100 hours last year building myself; I’m glad to see that Jack Clark (CEO of Anthropic) reports a similar feeling creating an app in between feeding his newborn baby. Bringing it back to Andrej Karpathy (who gave this talk which includes him reflecting on how his “shower thought” become a Wikipedia page) now at the end of the year:
I've never felt this much behind as a programmer. The profession is being dramatically refactored as the bits contributed by the programmer are increasingly sparse and between. I have a sense that I could be 10X more powerful if I just properly string together what has become available over the last ~year and a failure to claim the boost feels decidedly like skill issue. There's a new programmable layer of abstraction to master (in addition to the usual layers below) involving agents, subagents, their prompts, contexts, memory, modes, permissions, tools, plugins, skills, hooks, MCP, LSP, slash commands, workflows, IDE integrations, and a need to build an all-encompassing mental model for strengths and pitfalls of fundamentally stochastic, fallible, unintelligible and changing entities suddenly intermingled with what used to be good old fashioned engineering. Clearly some powerful alien tool was handed around except it comes with no manual and everyone has to figure out how to hold it and operate it, while the resulting magnitude 9 earthquake is rocking the profession. Roll up your sleeves to not fall behind.
5. Grok briefly (but very publicly) declares itself MechaHitler
As the video from 80,000 Hours is titled: If you remember one AI disaster, make it this one. “This is the postmortem that never happened, for the most deranged chatbot ever released.”
There was quite a lot that happened this year, but in many ways I think you can sum it all up — the world of tech, the world of US politics — with everyone’s favorite RollingStone headline:
On the “conversations” of 2025
1. Two major publications on what the future might look like… and both of them were inappropriately named
First, we had AI 2027, a well-developed reading experience of a highly speculative “choose your own adventure” future where the decision fork comes in the year 2027: you can chose to race ahead to beat China and everyone dies, or you can choose not to give the AI so much power… and (with the AI’s help) solve the alignment problem a few years later, ending with rainbows and ponies! They give detailed and data-driven explanations for their AI timelines and justification for thinking that AI takeoff will be in approximately 2027.
Very soon after the publication of AI 2027, Arvind Narayanan and Sayash Kapoor published “AI as Normal Technology.” Yes, they say, AI is a powerful technology, but a “normal” one, like smartphones or the internet. It is not going to completely upend society or cause the extinction of humanity or anything like that. The authors of AI 2027 wrote their response a little while later, predictably titled, “AI As Profoundly Abnormal Technology.”
Now, this is all well and good but… both of these papers were badly named!
The main author of “AI 2027,” Daniel Kokotajlo, spent the rest of the year saying that actually he thinks that the timeline will be somewhat later—and he even thought this at the time when they published it!
And the “Normal Technology” authors Narayanan and Kapoor later “clarified” that by “normal,” they didn’t mean “trivial,” in fact, their actual description of what they think the future will look like, depending on certain unknown factors, “would not be normal at all.” (You may have expected as much from authors of a book on “AI Snake Oil,” which despite the title, actually says that AI is likely to be one of the transformative technologies of the millennium).
2. To speak of the bubble
When tens of trillions of dollars — trillion, that’s $1,000,000,000,000 — move around, people start to take notice, and wonder: is this bubble gonna pop? Is the AI bubble actually inflating the whole economy?
There’s no point in linking to the dozens of articles on this question; obviously it is one that people care about with money on the line. Personally I liked this breakdown of the question (and how to think about the answer) by Derek Thomson, and I’ve found some especially good data-backed writing on this at FT (all of this is paywalled, sorry). This is particularly relevant in the world of AI-powered biotech, where it seems like there is much more money in generating hypotheses even as the bottlenecks to profitability are elsewhere.
I do not know whether there is an “AI bubble,” or even how to evaluate that question and/or whether or not we will be able to answer it in retrospect ten years from now. It does seem like there are a lot of risks that are hiding and not fully priced in, but hey, I’m not gonna short the market. All I know that this is now how people on the street talk about AI: are you on team bubble or no-bubble?
3. Gradual Disempowerment
It’s nice to have a good term to a thing that many people (myself included) thought was one of the most likely scenarios for catastrophic societal collapse due to AI, with a rigorous paper to back it up (with the associated webpage, of course):
This paper examines the systemic risks posed by incremental advancements in artificial intelligence, developing the concept of `gradual disempowerment’, in contrast to the abrupt takeover scenarios commonly discussed in AI safety. We analyze how even incremental improvements in AI capabilities can undermine human influence over large-scale systems that society depends on, including the economy, culture, and nation-states. As AI increasingly replaces human labor and cognition in these domains, it can weaken both explicit human control mechanisms (like voting and consumer choice) and the implicit alignments with human interests that often arise from societal systems’ reliance on human participation to function.
By the middle of the year, this topic was featured on the 80,000 Hours website with its dedicated “problem profile.” One of the authors gives a lecture version here; recently another author gave a podcast interview version with the Future of Life Institute.
As with the rest of these ideas, the concept very much existed before and continues to do so without the actual use of the term. In some ways (with some important differences), the idea of gradual loss of control due to being plugged into an artificial mind is behind one of the biggest TV shows of the year, PLUR1BUS. But long before 2025, the loss of control due to giving up your decision making skills to a superior intelligence was the subject of this very early Scott Alexander piece of online short fiction (and, in my opinion, still his most Borges-ian). The story got a recent shout out on X:
Oh, and there’s this, if you want to go for the “4chan aesthetic”: Durr chatGPT
4. The Jagged Frontier of AI Capabilities
Back in 2023, Ethan Mollick had a paper showing the improved efficiency of workers using AI, and referred to AI skill-level as “jagged:” they are better than humans at some tasks but much worse than humans at other tasks. I’m not sure how much influence this post had at the time, but this was the focus of Helen Toner in a much-discussed talk from October: what does it mean to take jaggedness seriously? And what if some capabilities shoot up to the moon, while AI remains very poor at some oddly specific tasks that humans can do?
5. Glazing, sycophancy, and psychosis
You could argue that these are different things, but there is one core problem at the root of it: LLMs are overly agreeable, to the point where they can reinforce users’ psychosis to the point where people become truly lost, sometimes in tragic and horrifying ways. This was completely predictable; if you train the model on user feedback, the model will learn to say things that cause the user to want to talk to it more whether these things are true and healthy or the opposite.
Besides for the tragic cases documented at the links above, this became a major concern in 2025 thanks to the rollout of GPT-4o, which was especially prone to ‘glazing,’ or being overly complementary of its human user (although, to be fair, Grok can be worse). Some examples were really hilarious.
It was so bad that OpenAI essentially wrote a long apology.
…and that’s a wrap! Here’s hoping for a happy 2026!
