The New Science of Alt Intelligence

The New Science of Alt Intelligence

For many decades, part of the premise behind AI was that artificial intelligence should take inspiration from natural intelligence. John McCarthy, one of the co-founders of AI, wrote groundbreaking papers on why AI needed common sense; Marvin Minsky, another of the field’s co-founders of AI wrote a book scouring the human mind for inspiration, and clues for how to build a better AI. Herb Simon won a Nobel Prize for behavioral economics. One of his key books was called Models of Thought, which aimed to explain “Newly developed computer languages express theories of mental processes, so that computers can then simulate the predicted human behavior.”

A large fraction of current AI researchers, or at least those currently in power, don’t (so far as I can tell) give a damn about any of this. Instead, the current focus is on what I will call (with thanks to Naveen Rao for the term) Alt Intelligence.

Alt Intelligence isn’t about building machines that solve problems in ways that have to do with human intelligence. It’s about using massive amounts of data – often derived from human behavior – as a substitute for intelligence. Right now, the predominant strand of work within Alt Intelligence is the idea of scaling. The notion that the bigger the system, the closer we come to true intelligence, maybe even consciousness.

There is nothing new, per se, about studying Alt Intelligence, but the hubris associated with it is.

I’ve seen signs for a while, in the dismissiveness with which the current AI superstars, and indeed vast segments of the whole field of AI, treat human cognition, ignoring and even ridiculing scholars in such fields as linguistics, cognitive psychology, anthropology, and philosophy.

But this morning I woke to a new reification, a Twitter thread that expresses, out loud, the Alt Intelligence creed, from Nando de Freitas, a brilliant high-level executive at DeepMind, Alphabet’s rightly-venerated AI wing, in a declaration that AI is “all about scale now”, Indeed, in his mind (perhaps deliberately expressed with vigor to be provocative), the harder challenges in AI are already solved (“The Game is Over!”, he declares:

There’s nothing wrong, per se, with pursuing Alt Intelligence.

Alt Intelligence represents an intuition (or more properly, a family of intutions) about how to build intelligent systems, and since nobody yet knows how to build any kind of system that matches the flexibility and resourcefulness of human intelligence, it’s certainly fair game for people to pursue multiple different hypotheses about how to get there. Nando de Freitas is about as in-your-face as possible about declaring that hypothesis, which we I will refer to as Scaling-Uber-Alles.

Of course, that name,, Scaling-Uber-Alles, is not entirely fair. De Freitas knows full well (as I will discuss below) that you can’t just make the models bigger and hope for success. People have been doing a lot of scaling lately, and achieved some great successes, but also run into some road blocks. Let’s take a dose reality, before going further, into how de Freitas faces facts.

Systems like DALL-E 2, GPT-3, Flamingo, and Gato appear to be mind-blowing, but nobody who has looked at them carefully would confuse them for human-intelligence.

DALL-E 2, for example, can often create fantastic artwork, from verbal descriptions:

But it’s also pretty prone to suprising stumbles, like this uncomprehending reply to “a red cube on top of blue cube”. [DALL-E’s responses, created by a network called unCLIP, are on the left of the figure; interestingly a predecessor model, shown on the right, actually does better on this particular item.]

When Ernest Davis, Scott Aaronson, and I probed in a bit more detail, we found a number of similar examples:

Similarly, the ostensibly amazing Flamingo has (not unrelated) bugs of its own), captured ina candid and important thread by DeepMind’s Murray Shanahan, with further examples Flamingo’s lead author. For example, Shanahan confronted Flamingo with this image:

and wound up having this discomprehending dialog:

DeepMind’s newest star, just unveiled, is Gato, is capable of cross-modal feats never seen before in AI, but still, when you look in the fine print, stuck in the same land of unreliability, moments of brilliance coupled with absolute discomprehension:

Of course, it’s not uncommon for defenders of deep learning to make the reasonable point that humans make errors, too.

But anyone who is candid will recognize that these kinds of errors reveal that something is, for now, deeply amiss. If either of my children routinely made errors like these, I would, no exaggeration, drop everything else I am doing, and bring them to the neurologist, immediately..

So let’s be honest: scaling hasn’t worked yet. But it might, or so de Freitas’s theory—a clear expression of the Zeitgeist—goes.

So how does de Freitas reconcile reality with ambition? Literally billions of dollars have been poured into Transformers, the underlying technology that drives GPT-3, Gato, and so many others; training data sets have expanded from megabytes to gigabytes, parameter counts from millions to trillions. And yet the discomprehending errors, well-documented in numerous works since 1988, remain.

To some (such as myself), the abiding problem of discomprehension might—despite the immense progress—signal the need for a fundamental rethink, such as the one that Davis and I offered in a our book Rebooting AI.

But not to de Freitas (nor to many others in the field—I don’t mean to single him out; I just think he has given prominent and explicit voice to what many are thinking).

In the opening thread he makes much of his view about reconciliation reality with current troubles clear, “It’s about making these models bigger, safer, compute efficient, faster at sampling, smarter memory, more modalities, INNOVATIVE DATA, on/offline”. Critically, not a word (except, perhaps “smarter memory”) is given to any idea from the cognitive psychology, linguistics, or philosophy.

The rest of de Freitas’ thread, follows suit. starting with the second tweet;

This is, again, a pure statement of scaling-über-alles, and marks its target; the ambition here is not just better AI, but AGI.

AGI, artificial general intelligence, is the community’s shorthand for AI that is at least as good, and resourceful, and wide-ranging, as human intelligence. The signature success of narrow artificial intelligence, and indeed alt intelligence, writ large, has been games, like Chess (DeepBlue owed nothing to human intelligence) and Go (AlphaGo similarly owned little to human intelligence). de Freitas has far more ambitious goals in mind, and, to his credit, he’s upfront about the,.

The means to his end? Again, in Tweet 2 of his thread, the main emphasis is mainly technical tools for accommodating bigger data sets. The idea that other ideas, eg from philosphy or cognitive science, might be important is dismissed. (“Philosophy about symbols isn’t” is a perhaps said rebuttal to my long-standing campaign to integrate symbol-manipulation into cognitive science and AI, recently resumed in Nautilus Magazine, though the argument is not fully spelled out. Responding briefly: His statement that “The idea that [neural] nets have no issues with symbols misses both history and reality. The history missed is the the fact that many neural net enthusiasts, have, for decades argued against symbols, and the reality that is missed is the above-documented fact that symbolic descriptions like “red cube on blue cube” still elude the state-of-the-art in 2022.)

de Freita’s Twitter manifesto closes with an approving reference to Rich Sutton’s famous white paper, The Bitter Lesson:

Sutton’s argument is that the only thing that has led to advances in AI is more data, computed over more effectively. Sutton is, in my view, only half right, almost correct with his account of the past, but dubious with his inductive prediction about the future.

So far, in most domains (not all, to be sure) Big Data has triumphed (for now) over careful knowledge engineering. But

Success on a few problems with a particular strategy in no way guarantees that we solve all problems in the similar way. It is sheer inductive folly to imagine otherwise, particular when the failure modes (unreliability, bizaare errors, failures in compositionality and discomprehension) have not changed since Fodor and Pinker pointed them out (in separate articles) in 1988.

Let me close by saying that I am heartened to see that, thankfully, scaling-über-alles isn’t fully consensus yet, even at DeepMind, who wrote earlier this morning:

I am fully with Murray Shanahan when he writes “I see very little in Gato to suggest that scaling alone will get us to human-level generalisation.”

Let us all encourage a field that is open-minded enough to work in multiple directions, without prematurely dismissing ideas that happen not yet to be fully developed. It may just be that the best path to artificial (general) intelligence isn’t through Alt Intelligence, after all.

As I put it on Twitter a few days ago, I am fine with thinking of Gato as an “Alt Intelligence”— an interesting exploration in alternative ways to build intelligence—but we need to take it in context: it doesn’t work like the brain, it doesn’t learn like a child, it doesn’t undestand language, it doesn’t align with human values, and it can’t be trusted with mission-critical tasks.

It may well be better than anything else we have currently got, but the fact that it still doesn’t really work, even after all the immense investments that have been made it, should give us pause.

And really, it should lead us back to where the founders of AI started. AI should certainly not be a slavish replica of human intelligence (which after all is flawed its own ways, saddled with lousy memory and cognitive bias). But it should look to human (and animal cognition) for clues. No, the Wright Brothers didn’t mimic birds, but they learned something from avian flight control. Knowing what to borrow and what not is likely to be more than half battle.

The bottom line is this; something that AI once cherished but has now forgotten: If we are to build AGI, we are going to need learn something from humans, and how they reason and understand the physical world and represent and acquire language and complex concepts.

It is sheer hubris to believe otherwise.

P.S. Please subscribe if you’d like to read more no-bullshit analysis of the current state of AI.

Images Powered by Shutterstock