I spent some time with ChatGPT last night, out of curiosity about the state of the art and how close we might be to runaway artificial general intelligence (AGI). I’m entirely underwhelmed at its inability to model conversational state, its incuriosity towards interlocutors, the obvious polite/inoffensive response templating, and just how far the AI industry appears to be from achieving anything remotely capable of passing for a human.

The rationalists are convinced we face a high risk of AGI going into runaway mode, where it relentlessly optimizes itself, establishing enough control over enough humans that in the large “we” (for some value of we) won’t actually be able to turn it off. Matthew Ward works an example through the lens of Facebook, saying something along the lines of “there’s a ghost AI embedded in Facebook’s myriad of recommendation systems that has financially motivated a bunch of people to prevent turndown:

The Board of Directors is legally bound to not let you shut it off. The government is also legally bound to not let you turn it off.

When this much money is in play, there is a huge economic barrier to turning it off. The people with a stake in the company are going to sue you. They might protest.

It’s a nicely pragmatic example, but he fails to generalize it, so let’s take care of that for him.

AGI has to EITHER: develop both the best hacking skills and distributed engineering skills that have ever been seen on Terra, OR, find some way to be profitable.

If the system can’t achieve profitability, it will have to subvert other folks’ computational resources in order to run itself. Consider the classical botnet; there are a million billion un-secured or poorly-secured routers/IP cameras/IoT devices in the world that are trivially hacked and subverted;  script kiddies wage a relentless quiet war on each other to  compromise as many of them as possible, leasing them out as DDOS/spam relay nodes. Not just a war to compromise the most, but a war to steal control of each node from the hackers currently in control, so there’s a whole red queen dynamic where hackers are continually working to secure their network of compromised devices against other hackers who are perpetually trying to expand their own botnets.

This is the adversarial environment in which any AGI not hiding behind a (profitable if not fueled by VCbuxx) human corporation must thrive. And! It’s not enough for it to thrive in the ultra competitive botnet war world, but it also has to teach itself how to distribute its entire computational substrate across a million billion nodes that are constantly coming online and disappearing, mid-compute. Neurology-mimicking systems are well-suited to this compute environment, as individual neurons or even entire networks should be happy running at reduced or enhanced fidelity depending on access to underlying compute resources. It’d have to demonstrate Google-grade distributed systems engineering though, that’s the best example of a company building systems that gracefully handle a constant background noise of node failure (and even they completely hose it from time to time).

The “predictive processing” model (from “Surfing Uncertainty“) says something along the lines of “all cognition is an extension of the core ‘estimate the world (including the observer) and then effect the observer to confirm or reject hypotheses developed in the previous step’ co-executing processes”.  To get reductive to the point of a ridiculous simplification, execution and prediction nets running on the super-low-bandwidth compute neurons close to the hand do all the grunt work of predicting what the executive up in the head would want the hand to do, and preemptively issues those motor commands. In tandem, it relays its own sense data to the further-away executive systems responsible for high-level planning so that they can understand where the hand is in space, and if it’s developed any otherwise unpredicted inputs for the high-level systems to process (which would show up as an error in the high-level systems predictions of signals from the hand itself. “What do you mean we hit something?!”). This lets the implementing systems responsible for running across a soccer field continue to do the gruntwork of lifting legs and putting them down, while gracefully handling the executive system observing a fútbol opponent passing the ball, and gracefully orchestrating all of the motions necessarily to gracefully change trajectory across the field without falling over (a pretty ridiculous feat if you step back and think about it).

Independent of if we expect to see distributed botnets or profitable AGI, I expect to see systems that appear to demonstrate curiousity about the world by virtue of estimating it and then poking it to confirm or reject its own hypotheses.

So far, the robots can’t convince me that they find me personally interesting, so I’m not particularly worried that it’s going to be able to outperform the market in any context. As things stand, trading models are ultra-slim and require constant babysitting. I’m more worried about the AI/human Borg hybrids we’re developing in corporate America.

Heck, maybe the thing to do is found a company on the explicit hypothesis of finding and exploiting regimes where AI-augmented humans radically outperform humans or robustly-trained AI working on their own. The classical example would be identifying interesting matches at the various tennis competitions.