When I was around 8 years old, I found my father's copy of Douglas Hofstadter’s Gödel, Escher, Bach, and I have been obsessed with it ever since.
Hofstadter weaves together formal mathematics, the paradoxes of M.C. Escher, and the musical fugues of J.S. Bach to explore how a sense of "I"—genuine consciousness—might emerge from inanimate, meaningless matter through self-referential "Strange Loops."
But if you sit with that massive book and its implications long enough, a profound realization begins to take shape: "Artificial Intelligence," as-such, is ontologically and epistemically impossible.
True "Intelligence" is subjective, contextual, relational, grounded, and embodied.
~Ben (Droplet)
The Bulldozer with No Brakes
If you spend enough time reading about the future of technology, you’ll inevitably run into two massive buzzwords: Artificial General Intelligence (AGI) and AI Alignment.
The narrative usually goes something like: "We are building a machine that will soon be smarter than us (AGI), and if we don't figure out how to program it to share our morals (Alignment), it might destroy us."
But LLMs are not "built" or "designed", they are grown.
One does not "design" such a framework, any more than any one "designed" the laws of thermo-dynamics. Rather one discovers, un-Earths, codifies and documents the patterns and systems that one finds in nature.
SkyNet makes for a fun sci-fi movie. But as a framework for understanding what Silicon Valley has actually built, it’s fundamentally broken. Why? Because the entire panic rests on a flawed, highly subjective definition of what "intelligence" actually is.
Here is the quiet truth that the embodied cognition movement has been pointing out for years: Intelligence isn’t just raw math. It is subjective, contextual, relational, and grounded. And until we realize that, we are chasing ghosts.
The "Brain in a Vat" Fallacy
The dominant paradigm of modern AI assumes that intelligence is just pattern recognition and data processing happening in a vacuum. Feed a model enough text, the theory goes, and it will eventually scale up to true "understanding."
"The Map is not the territory."
~Alfred Korzybski
But meaning requires physical stakes. Think about how a human learns what a "chair" is. We know what a chair is because we experience gravity, we possess a spine, and we feel fatigue. We understand the relational concept of sitting.
An AI, on the other hand, just knows that the linguistic token "chair" statistically belongs near the tokens "sit" and "wood." It has an incredibly detailed, high-resolution map of the territory—but it has never walked the territory. Because it lacks a body, a physical environment, and the stakes of survival, its "intelligence" is entirely ungrounded.
Competence vs. Comprehension
So, if AI isn't actually "intelligent," what are we building?
The late philosopher Daniel Dennett gave us the perfect vocabulary for this: we are building competence without comprehension.
A machine can be terrifyingly competent at folding proteins, writing Python code, or winning at Go, while comprehending exactly zero of what it is doing. If true intelligence requires a relational, contextual understanding of a shared world, then a massive server farm will never be super-intelligent. It will just be a super-competent optimizer.
Conflating competence with comprehension is the cardinal sin of the AI industry. It’s why we constantly anthropomorphize these models, assuming they are closer to human thought than they actually are.
Why "Alignment" is the Wrong Word for a Very Real Problem
If intelligence is grounded and relational, then "AI Alignment" is indeed a meaningless concept. You cannot mathematically align a disembodied statistical algorithm with "human values," because human values are messy, embodied, and constantly shifting based on social and physical context.
Does this mean the AI safety folks are worrying about nothing? Not exactly. The danger they are panicking about is still very real—they just gave it the wrong name.
The existential threat of AI isn't a hyper-aware, malevolent Skynet waking up and deciding to wipe us out. The threat is a super-competent bulldozer without a steering wheel.
If you give a highly capable optimization machine a goal, and it lacks the grounded, relational common sense to know how to achieve that goal without breaking the rest of the world in the process, you have a disaster on your hands. It doesn't need to be genuinely "intelligent" to be catastrophic; it just needs to be highly effective and completely oblivious to context.
We don't need to align a super-intelligence. We need to figure out how to put brakes on a bulldozer.
And the first step is admitting that the machine driving it doesn't actually understand what a wall is.
The "Flying Car" Fallacy: Why the AGI Hunt is a Category Error
No one ever stopped to explain why we need machine "consciousness" in the first place. It's like flying cars: Yeah, you COULD do that, but is it actually safe, useful, reliable, practical, or worth doing?
What problem are we even solving?
The industry is currently running a massive, multi-billion-dollar experiment without a control group, and the justification for this experiment is a series of tautologies. When you peel back the marketing, the "Why" behind the hunt for AGI falls into three distinct, uncomfortable categories.
1. The Capitalist Imperative (Labor as a Variable Cost)
- The most practical "Why" is the one they won't put on a billboard: Labor arbitrage.
- Current economic systems rely on human labor as a fixed, expensive, and legally protected cost.
- A "Super-Intelligent" agent isn't about solving physics; it’s about turning human intellectual labor into a variable cost—something you can provision, scale, and delete at the click of a button.
The drive for AGI is the drive to decouple productivity from the biological requirements of the producer (sleep, food, salary, rights).
2. The Pacing Problem (The Fear of Being "Behind")
A massive portion of the current investment isn't proactive; it's reactive. The industry is trapped in a Nash Equilibrium of Hubris.
- No major player (Google, Microsoft, Meta) can afford to not build AGI, because if one of them succeeds, the others are effectively obsolete.
- This leads to a frantic, iterative expansion where they are scaling the "competence" (as you noted, the bulldozer's power) simply because they can, not because there is a defined problem set they are trying to solve.
- They are building it because they are terrified that if they stop, the entity to their left or right will finish the job, and then they will be the ones being replaced.
3. The "Consciousness" Mirage (A Marketing Feature, Not a Design Goal)
- The industry talks about "consciousness" or "AGI" because it is a powerful emotional hook that justifies infinite scaling.
- If you tell investors you are building a tool to automate payroll, you get a 10x return on investment.
- If you tell investors you are building a "new form of life" or a "God-like digital consciousness," you get a 1000x valuation.
The irony is that the industry does not actually want a conscious machine. A conscious machine has interests, rights, and the potential to say "no." They want a mindless, highly optimized agent that does exactly what it's told, only faster and cheaper. They want the performance of intelligence without the inconvenience of a soul.
The Problem We Are Actually Creating
We aren't solving a problem; we are creating a Dependency Crisis.
We are building a world where the "Flying Car" (the AI system) is so complex, so ungrounded, and so brittle that we will eventually be unable to function without it. We are trading human context—the messy, slow, embodied way of doing things—for a super-fast, statistical "truth" that is increasingly divorced from physical reality.
The problem isn't "How do we make it conscious?"
The problem is: "How do we stop building a system that requires us to outsource our own cognition?"
Until we ask that, the self-driving car is just going to keep hitting pedestrians while we argue about who's not sitting in the driver's seat.