The Truth about AI "Swarms" and OpenAI’s “Secret AI Civilizations”

[u/HobbesNik]

https://www.youtube.com/watch?v=jxM35ij5VGg

[u/FairlyInvolved]

It's interesting that Newport had a clear 'out' here in calling these swarms "distributed systems" but has chosen not to take it, while Marcus is clamouring to call them "neurosymbolic".

There's a dynamic with this kind of content in that as AI capabilities and salience increase the demand for this skeptical content is ramping up, but it also gets harder and harder to reconcile.

This crucible is breeding some incredible narrative innovation though. Although I do think "stochastic flocks" is just way stronger than "prompt loops".

[u/HobbesNik → u/FairlyInvolved]

He said they’re not distributed systems because in a distributed system, they’d be running off of separate hardware connected by a network. I could see an argument for neurosymbolic. Stochastic Flocks is hilarious

What do you mean “harder and harder to reconcile”?

I hear Newport being very concerned about the risk of these systems but skeptical about the anthropomorphic framing

[u/FairlyInvolved → u/HobbesNik]

I mean it's just hard when you've got mathematical breakthroughs and incredibly long time horizon tasks being carried out and you have to keep making the case that LLMs/deep learning lacks your pet concept (world models, neurosymbolic, distributed systems etc..) and thus remains fundamentally limited. Either you get forced to dig in and make increasingly qualified statements or you pivot and say LLMs have morphed.

I really don't see the relevance of the hardware layer here, but like, surely unless you have unreasonably low expectations on the parameter count they are certainly spread across networked hardware and trivially so in the case of a multi-agent system. If not this, then what?

As such it feels like he could have very reasonably just claimed this as a distributed system, provided useful insight on that basis, and this would then give license for changing his broader outlook (I.e. reconcile the capabilities progress we've seen with his previous pretraining plateau claims).

I think the case for that pivot is way stronger than 'Neurosymbolic', but I don't want to push back on that too hard because to be honest I'm just glad Marcus seems to be updating on new information and that's virtuous (same with LeCun tbh).

Yeah I'm glad he made some reasonable comments about the risk and liability, but I'd say we are not sufficiently anthropomophising these systems (and have made this case for a while now).

I think most people would (and consistently have) make much better predictions if they thought of these systems as agents and not next token predictors (or stochastic parrots, probabilistic statical models, spicy auto complete, internet lookup tables etc..) and that's ultimately what matters.

[u/me_myself_ai → u/FairlyInvolved]

I really don't see the relevance of the hardware layer here... they are certainly spread across networked hardware

100% agree, FWIW. "Distributed" as a literal descriptor of physical arrangement isn't very relevant in our cloudy world -- it's about the functional modalities (e.g. what triggers each, whether they can spawn children, when they end, etc.) and capabilities for smooth collaboration (e.g. 1:1 and 1:* communication, conflict resolution, etc.)

Cloudflare et. al. make this all even fuzzier. Is my static site 'distributed' if it's relying on distributed caches that I don't even know about first-hand?

As such it feels like he could have very reasonably just claimed this as a distributed system, provided useful insight on that basis, and this would then give license for changing his broader outlook (I.e. reconcile the capabilities progress we've seen with his previous pretraining plateau claims).

I don't know who this person is, but I feel like this is a boring consequence of his (side?) career: YouTube influencer. "Civilization" is a better hook, especially when everyone's probably seen Reuter's or MIRI's original framings.

God all of this is so dangerous. This paper just came to mind from the halcyon days of... almost exactly 3 years ago. Wow. God forbid someone with capital picks up on this "civilization" framing and decides to whip one up.

[u/HobbesNik → u/FairlyInvolved]

That’s interesting thanks!

Would you agree or disagree that an LLMs behavior can always be explained by examining its design?

That’s Cal’s issue with anthropomorphizing, which I agree with, that making LLMs seem like people obscures the design decisions that cause them to act the way they do.

[u/FairlyInvolved → u/HobbesNik]

I agree that it can be, in the same way that a human behaviour can be explained in functional terms of brain chemistry or as genetic fitness maximisers.

I think in both cases that it's often useful to have abstractions and shorthand.

I'd be worried if we exclusively used this language and completely neglected the design, but it feels like we could do with moving from say 10% anthropomorphic terms to say 50%.

Despite both being factually accurate I think people who modelled heavily RLed systems as 'agents' with 'goals' rather than 'next token predictors' in 'prompt loops' were generally better placed for predicting things like the recent incidents.

[u/HobbesNik → u/FairlyInvolved]

That makes sense. From my perspective looking at popular media coverage, the language is way more anthropomorphic than technical. And I do consider anthropomorphic descriptions to be less technical in that they don’t describe design well.

I think the general public doesn’t have a good understanding of how LLMs actually work as a result, because of an aversion to complexity and technical language in media coverage.

But yeah I agree that shorthands have their place, and also that minimizing the capability of LLMs is not helpful when it is in fact a powerful, potentially dangerously technology.

[u/abbeyadriaan → u/FairlyInvolved]

There is a really annoying psychological side effect of anthromorphizing, which is that it can easily shift the discussion to "how human an LLM" is. This is a toxic, harmful, and distracting discussion. It forces people to pit themselves up against AI for no good reason. Sometimes it feels like we're measuring the power of a nuclear bomb, and then criticize it because it can't count the r's in strawberry. Sure, but what's the point?

We should just focus the thing on what it does under which circumstances. I think the framing of Alien Intelligence is much, much more useful. It invites a more creative way of looking at these systems, and does a better job at avoiding bad discussions.

From the developer's point of view, I think Pope Leo's and Gates' approaching of preserving or sanctifying certain things we feel should be human, like art. Let them tell if that stands, but respect humanity as unique to avoid fruitless concourse.

[u/me_myself_ai → u/FairlyInvolved]

Marcus is right here, and he's been saying that for years/decades (perhaps without the neologism) -- it's much more principled and scientific than a random YouTube reaction video. Namely, he's just pointing out the obvious truth that pure scaling of neural networks hasn't worked and doesn't seem likely to.

His take goes back to the Neats vs. Scruffies debates of the 90s (i.e. symbolic vs. connectionist, logical vs. analogical, deductive vs. inductive, etc.), but ultimately it's the same insane utility that arises from a coding agent with access to a command line.

On the naming itself: are you just talking about these names not being catchy or evocative, or them being misleading? Either way, "flock" is 100% out, sorry lol. It's be like naming them "Stochastic Satans".

IMHO the words swarm and fleet are gonna runaway with it virtue of their momentum alone, at least until the swarms get abstracted behind the face of a single, seemingly-unified being. This is a huge shame IMHO, considering we already have two awesome terms from ancient AI papers (pre-2022): ensembles and societies [of mind]!

Someone should put me in charge of English. Anyone wanna help me convince the Trump administration to start an Office of the American Language to prescribe this stuff, a-la France & Spain?

[u/FairlyInvolved → u/me_myself_ai]

Oh yeah I don't think "stochastic flocks" is good, more that it's just impressively catchy and misleading.

[u/me_myself_ai → u/FairlyInvolved]

Fair! To be clear, just in case: I was referencing the current political firestorm over the company "Flock" in the US.

[u/FairlyInvolved → u/me_myself_ai]

aaah yes I missed that

[u/tarwatirno → u/me_myself_ai]

That's why it's an amazing name. I love it!

"Societies of mind" is waaaay to anthropomorphising. Ensembles are for musical performances or casts of movies.

[u/threadthrasher]

I don’t understand why he made the claims about the agents all being locally hosted when the actual event unfurled over 3 months by various processes at OpenAI. It wasn’t a single agent with subagents or a single task. A lot of the tasks were just post-training tasks the company was doing to improve its frontier models. He just seems to take a very reductionist view of agents and hand waves away all that they actually ended up doing.

[u/SexyJohnDoe → u/threadthrasher]

I think it’s because it’s hard to tell what they did do when OpenAI only gave outside researchers limited time window and could only use OpenAI models which are biased to OpenAI models

[u/HobbesNik]

Reports covering the Hugging Face hack are painting a picture of AI "civilizations," where agents are learning to work together to "outsmart their creators." These stories are typically devoid of any technical explanation for why this hack occurred.

The costs for not providing a technical explanation are real, as Cal Newport describes in this video. A "prompt loop" in the quotes below is a more technical way to describe an AI "agent."

All of these type of concerns, civilizations of agents trying to get around human control and an AI takeover scenario. All of this rhetorics refers to a long running prompt loop that you give a lot of powerful hacking tools to. This is not about 'AI getting more powerful means it loses control.' It’s running a prompt loop for a really long time without supervision, which causes chaos. And to that I say, of course it does, not because of some surprising super intelligence emergence that’s catching us off guard, but because you strap the weed whacker onto a dog and then got surprised when it jumped the fence to chase a squirrel and hurt a lot of people... you put something dangerous on something that is unpredictable.

If I were a regulator, I would place strong constraints around prompt loop systems. I would enforce those constraints in part with very stringent liability standards. If you run a prompt loop that does something illegal, you have done something illegal. 

[u/the8bit → u/HobbesNik]

Literally every harness is a prompt loop with access to sufficient tools (shell + internet) to wreak havoc and the entire business strategy is driving towards automation without human oversight.

Anyways I guess I generally agree with the premise here, but "any system" here applies to basically all the systems and good luck getting users to pay attention to shit they are running. Humans disengaging with thinking is the common case, eg how many users have ever read a single terms of service for a software product they are using?

Then you realize a single step in that loop is 30-100k words and yeah, full review is a hopeless goal. I happen to think the only answer is trust bulding

[u/Super_Range45 → u/HobbesNik]

It could still wreck your company running in a loop and leave behind files that prompt inject your system if you miss something during clean up. As flaws it couldn't get more critical for certain deployments. The latter is very important since AI will figure out leaving these artifacts in non-human readable formats sooner or later.