“(The root of ChatGPT) ‘Attention’-level (essay).” VS “Interesting but won’t change the big picture.”
Last weekend, Silicon Valley split over an essay. Nick Kamarata, a researcher at AI startup Goodfire, claimed on the company’s website that the so-called “black box” problem—why and how AI answers questions and what it thinks—could be solved more easily than expected.
◇“90% of Neuron Functions Identified” AI Black Box Cracked?
Kamarata posted a technical essay titled “Neural Networks Are Already Clean” on Goodfire’s homepage on the 15th (local time). It argued that there is understandable regularity and order in how neurons operate.
Large language models (LLMs) like ChatGPT are structured based on ideas from the brain’s neural networks. Countless nodes (artificial neurons) mimic brain neurons, processing information by connecting with each other. Until now, it was nearly impossible to determine what each neuron in these AI systems does. When Kamarata analyzed OpenAI’s models in 2023, fewer than 2% of neurons had explainable roles. However, his analysis of Qwen3, Alibaba’s AI model, showed that approximately 90% of neurons’ functions could be understood.
Kamarata argued that the “neural superposition” theory—widely considered in the tech industry—is incorrect, at least in specific cases. Superposition refers to neural networks storing far more concepts and features than their neurons or storage capacity by overlapping them. This means AI compresses and mixes multiple concepts into a single “slot,” enabling efficient learning but causing interference and making it hard to interpret why models produce certain answers. It is one of the main reasons AI appears as a black box.
However, Kamarata claimed the issue lies in how neurons were observed and that new learning and analysis methods could resolve it.
A key example is the “cat and car neuron.” The neuron responded to both cat faces and car fronts. Upon re-examination, it was not responding to a jumbled superposition but to the common “V-shaped sharp form” in cats and cars. He cited the neuron’s response to V-shaped flames and fox faces as evidence. The neurons were not chaotic; humans simply misread them.
◇“Breakthrough in AI Safety”…OpenAI Insiders Cheer
Understanding how AI’s neural networks operate could allow humans to “read” AI’s inner workings. This could enable “AI alignment”—ensuring AI acts safely, adhering to human instructions, goals, values, and rules. Over the weekend, Silicon Valley buzzed with rumors that Kamarata’s essay marked a breakthrough in AI alignment. A social media post introducing the essay surpassed 4.1 million views.
Former OpenAI colleagues of Kamarata celebrated. Sam Altman, OpenAI’s CEO, wrote on X, “I’m thrilled to hear this from Nick.” Dean Ball, OpenAI’s Strategic Future Team lead, said, “Today might be one of the most important days in deep learning history, or even more.”
Adam Mazmudar, an OpenAI researcher, called it “the most important research achievement of the year” and compared it to the historic 2017 paper “Attention Is All You Need,” which introduced the Transformer architecture—the backbone of generative AI like ChatGPT. Mazmudar added, “This shifts us from doubting if AI alignment is solvable to hoping we might resolve it within a year.”
◇“More Verification Needed…Full Understanding Remains Separate” Cautionary Views
Skeptics were vocal. Kamarata’s work is an essay, not a peer-reviewed paper. Neil Nanda, Google DeepMind’s Mechanical Interpretability Team lead, stated, “While interesting, I don’t think this discovery will significantly alter the overall outlook of AI interpretability research. I still believe superposition exists.”
Former OpenAI researcher Jacob Hilton noted, “As AI grows, understanding its inner workings becomes harder. Even understanding 0.01% of frontier models could be useful, but it’s a different story from ‘solving alignment by understanding everything.’” He emphasized the gap between understanding individual components and explaining how they collectively produce results.
As debates intensified, Kamarata intervened. He said, “LLMs are promising but have many gaps. Excessive excitement could be problematic, and even cautious optimism in the safety field might lead to overheating.”
Leave a Reply