Why The Recent Claims Of AGI Fail: Plus, A New Test For AGI
The Gödel Test
There are two new claims from last week that artificial general intelligence (AGI) has been reached, as detailed in Tree of Woe’s recent article on the subject. One claim is from an AI investment firm (of some sort), and the other by some academics.
Ed Feser beat me to an essay investigating the academics’ claim: “No, AI does not have human-level intelligence“. I’ll take a different tack criticizing the investment firm’s claim. I’ll take it for granted that all of you will first read Feser’s article, in which he explains how the loose hazy wishful (my word) definition of intelligence used by the academics is inadequate. I’ll assume you know what he has said on this below. For instance:
Having failed to provide a serious definition of intelligence, it is no surprise that they also fail to provide a serious account of how to go about detecting intelligence (since the latter task presupposes the former). There is a lot of hand waving about “a cascade of evidence,” and gee-whiz references to what LLMs can do. But all of this ultimately boils down to nothing more than a stale appeal to the Turing test. And the problem with the Turing test is that, of its very nature, it cannot distinguish genuine intelligence from a mere clever simulation. Indeed, it deliberately ignores the difference and focuses narrowly on the question of what would lead us to judge a machine to be intelligent, rather than the question of what would make it the case that a machine actually is intelligent.
Ed’s critique is philosophical, of course, and follows in part things like this: “The Limitations Of AI: General Or Real Intelligence Is Not Possible” (in which we go over Searle’s works). But philosophical arguments will not be, though they ought to be, convincing to those excited about AI. These folks hope that if they throw enough RAM and electricity at the problem, the Impossible Gaps from mere code to genuine intelligence can be bridged in a way that nobody has yet figured. “Emergence” will kick in and the computers will come alive.
My critique is not philosophical, but practical, from someone who writes code for a living, and from my thinking and experience with models.
At the end, I propose a test for AGI. Incomplete. Those interested in this subject can help refine this test.
My Critique
AI is models. Models do not parrot, or simply repeat back what they have coded to regard as history. They make predictions. Take this simple “regression” model pictured here. The data is in XXX, also fabulously called “the training data”, as if models can be trained like dogs. The data (the plot) reads “When X takes this value, Y took these values.”
The line is the model itself, a simple mathematical function of the observations. The line smooths over the observations, and simply states “When X takes this value, Y takes that value.” Not values, but here singular. It is then simplicity itself to propose any X and see the model’s corresponding Y. More sophisticated models might even state the uncertainty in the Y, given this X.
The X can be, as scientists thrill to say, “novel”, i.e. an X never before seen. Yet we can still produce a Y for it, as the picture shows. The vertical line is an X never before seen by the model. Its prediction is in red on the line (of course), and the Y real life gave us is in green.
When a new, or any X, is given the model, the model gets something done. It produces work. It may even give a good response, in the sense of Y’s closeness to Reality when Nature herself takes this never-before-seen X, and churns out a Y through whatever causes are required.
But this regression is clearly a model. It is math, nothing more. It is easy to see this, too, since it is compact. No one would say the model is alive, has reached sentience, has consciousness or somehow developed an intellect and will. Its just dumb code. Speechless code. Mute. Incapable of language. The code has not reached self-awareness.
Yet some will say the code has “learned”. As a metaphor, the word has its uses. But it is wrong. Strictly incorrect. There is nothing in code that can learn. It’s just code! The interpretation of it, the meaning, comes from us, from outside the computer.
We can also use that same distressing metaphor to say that the model has “figured out” the Y with our “novel” X. The real figuring came in the persons doing the coding, the code itself is just a machine. But, as unhelpful and misleading a this metaphor is, it is not wrong to say that the model has “figured” a new thing.
But it did so without intelligence. Novelty, in this simple sense, is thus not intelligence.
Our simple model can be juiced in many ways. For instance, by writing code that fetches new data, which searches are triggered by any number of flags in the code. The model can redo its own regression, coming up with better, tighter guesses of Reality. We can have it flag certain correlations, and task it to make new models like that regression, all inside the larger Meta Model.
The whole thing can be automated. Somebody pushes Go and off it runs, with no hands required to keep the thing rolling along. As long as nobody unplugs it, it can keep running and producing new mini-models (the new regressions).
But none of this is intelligence. Not even if we add more code that makes the model spit out English to explain its results. Not even if the eventual mini-models have never been thought of by the coders of the AI. This is still only code plugging along, doing what was told, because—all regular readers know this chant—all models only say what they are told to say, and AI is a model.
We—the collective we—know what the code is in all AI, because it is we who write it. That the output, produced by myriad combinations of the inputs (always newly arriving through user interaction) with the manipulations given by the coders, is unpredictable to the coders does not mean that the code was intelligent. It only means the intelligence of the coders was limited, because their creation is too hugeous. Too many knobs, switches, and dials, the combinations of which grow fast. Complexity of output is not intelligence.
There will be many claims of AGI. Most will be sincere, but, since money is on the line, a lot will be pure hype.
Here’s Sequoia’s claim:
A Functional Definition of AGI
AGI is the ability to figure things out. That’s it.*
*We appreciate that such an imprecise definition will not settle any philosophical debates. Pragmatically speaking, what do you want if you’re trying to get something done? An AI that can just figure stuff out. How it happens is of less concern than the fact that it happens.
A human who can figure things out has some baseline knowledge, the ability to reason over that knowledge, and the ability to iterate their way to the answer.
An AI that can figure things out has some baseline knowledge (pre-training), the ability to reason over that knowledge (inference-time compute), and the ability to iterate its way to the answer (long-horizon agents).
Some words follow about what an AI agent is (not important for us) and then this:
This is what it means to figure things out. Navigating ambiguity to accomplish a goal – forming hypotheses, testing them, hitting dead ends, and pivoting until something clicks. The agent didn’t follow a script. It ran the same loop a great recruiter runs in their head, except it did it tirelessly in 31 minutes, without being told how.
To be clear: agents still fail. They hallucinate, lose context, and sometimes charge confidently down exactly the wrong path. But the trajectory is unmistakable, and the failures are increasingly fixable.
There is a great deal of confusion in all this, except they’re right that none of this will settle any debates.
We have already seen that models “figure things out” in a metaphorical sense. This does not make them intelligent. We have also seen that we can code Meta Models to form mini-models, which coded actions if you want to call, metaphorically, “forming hypotheses” you can. But this isn’t intelligence.
The Meta Model is following a script, though, and the claim that it isn’t is a bluff. Of course it is following a script. That is what code is. Scripts. It is a mere label to call the Meta Model an “agent” that builds the mini-models.
Models do not hallucinate. See my “AI Cannot Hallucinate Nor Lie“. Models make inaccurate predictions (when they do not know the full causes and conditions of what it is they are predicting.)
It is well here to remind ourselves of the awful nasty habit computer scientists have in giving false misleading grandiose names to their packets of code. They simply cannot break themselves of this debilitating tic. Problem is, these inaccurate names, cute in one sense, become like sycophantic Yes Men, which leads the computer scientists to start believing their own press.
You’ve seen the effects of this in politicians and celebrities: the artificial worlds of hype and exaggeration they build for themselves become realer than Reality. Computer scientists are constantly jumping ahead of where they are, because the names they give their creations impel them. Consider the actual example of touting “A new non-linear regression model”, and then seeing how many more investors you can tease up by renaming it “Neural nets”. (Yes, this really happened.)
Finally, even though these critiques, the philosophical and practical, may be convincing for the moment, their effects will wear off. Too much competition from the glee and constant promotion of the AI “revolution.” (And in genuine modeling successes, like this.) It would be best if some kind of test of AGI were in hand, agreed upon by critics like Feser and myself, and by AI mavens and promoters. Some practical task which would hack through the hype and restore calm.
AGI Test
A rational intellect knows its own mind, knows it knows it, and wills what it thinks is good for it. That happens in man. How could we tell if happened in a machine? Not from “metrics” like “getting things done” or “solving problems”. Any soulless model does these things, as we have seen. How can we determine a “souled” model? Not by the “Turing test”. The goofy ELIZA chat program was fooling people computers had intelligence in the 1960s, passing Turing’s test with ease.
We’ve already seen, too, that Meta Models can build mini-models, with neither being intelligent. Too, complexity of output is no guarantee of intelligence. Cut open a ten pound bag of rice and scatter the contents wildly. The resulting pattern of strewn grain will be exceedingly complex, but there is nothing intelligent in it. Which is a reminder that it is we who bring our intelligence to bear when examining computer output. It is we who give those patterns (and lack of patterns!) meaning. To the computer, they are just electrical impulses, no one set being different than another in meaning.
You can input textbooks on logical syllogisms, and then have the AI model predict new ones, using new propositions/premises as input. But that does not mean the model has grasped the meaning of the syllogisms. But this also means that a test where a model spits out new mathematical theorems or syllogisms, that turn out true, would be inadequate. The trivial regression model above can do as much. You can’t distinguish between genuine intelligence and mimicry (Ed’s word) here by accurate predictions.
So here is one idea, not yet well thought out or complete. Please help if you can.
One thing man does it sit and contemplate. In-sights are gained in this way (and other ways, too, of course). When we are quiet, undistracted, and not even necessarily thinking of the subject at hand, in comes the realization, knowledge of a new universal truth.
Every serious student has had the experience when thinking hard on some subject of making a discovery, true discovering, but ones which (sadly) we learn have already been thought up by others long before. The precedance is only important in the race for credit (taking into account Stigler’s Law of Eponymy, of course). What we have done is still an instance of genuine intelligence. Every teacher knows this.
This suggests a test for AGI. Have the model “trained” only on an agreed upon set of elementary mathematics. Perhaps up to high school algebra. No hint of complex numbers or anything “advanced”. The Meta Model to manipulate the “training” into new objects can be built as usual.
Then close off all new input. Let the model contemplate. We only allow it to “ponder” what it has. We have already seen it will be able to make accurate predictions at times. But these will only be of the same level or order of the material it already has. Polynomials of astonishing complexity would be proof only that the model has no bugs. They would not be proof the model has demonstrated intelligence.
But coming up with complex numbers would. Deriving the calculus from principles it had no way of knowing, and could not be built from simple premises would. Discovering measure theory and Cantor’s Paradise would. These all require leaps of inductive intellection, and are a clean a test of intelligence as you can find.
I had thought of calling this the Gödel Test, but somebody beat me to it. However, their test is easier, only having to solve conjectures, which is no different than making predictions. We’re asking for more than answers to conjectures. We’re asking models to rediscover insights.
The name is still good, because in essence, or if you like, Gödel proved new axioms, i.e. these acts of inductive intellection, are necessary to move into new areas of thought. We’re asking the model to create entirely new, to it, areas of thought, which require what we can be sure it did not have in its code to begin with.
That’s only a sketch. We have to be incredibly cautious about the setup, because it’s too easy to fool ourselves, or to redefine success after the fact, as this sage video demonstrates.
Here are the various ways to support this work:
Way back in the foggy mists of time, about 1964, I had my first job as a computer programmer. Some evenings a few of us would gather in the glass walled room with the big starship console, the big gray memory cabinets, the card reader, the printer, and two walls of high speed mag tape units. The operator would mount a tape and push a button. The tape units and printer would come to life. Instructions and status messages would type themselves on typewriter terminals located around the room. It was kind of spooky, as though the computer was alive. The press called computers "electronic brains,” and sci-fi became obsessed with visions of computers becoming gods or demons. There was all kinds of speculation about whether computers were or could become smarter than humans. Most of those doing the speculating then, as now, had little or no experience with actual intelligence and certainly couldn't define what it was or what it could do. The scammy idea of Artificial Intelligence evolved, and every few years some Professor Feigenbaum or Minsky etc would have an AI flavored bridge to sell us. AI is simulated intelligence, not artificial intelligence. Successful feats of automation or computation are often misrepresented as AI to fool the investors into thinking that AI has become skilled and useful at something. The current AI bubble is already deflating because it has reduced itself to absurdity by demanding that the electric grid be quadrupled in capacity to “train” LLMs. Woe to all the electric car drivers!
It’s interesting that the delusion of intelligence that the VC and Bro-ligarchs promote is now accompanied by the demand for access to ALL materials produced by humans. Because no one would want to prevent the creation of a new form of life, nor stand in the way of the Manhattan 2.0 project.