Rendered at 18:23:33 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
Zigurd 4 hours ago [-]
LLMs, plus broadly sourced yet expertly curated training sources, plus clever harnesses, plus RAS, etc. do an ever better job of synthesizing their training set into useful responses. For some use cases like coding, that's very useful now and likely to get at least somewhat better before reaching limitations based on the training set.
That's not going to reach AGI, mainly because today's recipe for AI products isn't built to be AGI. Some people believe it will reach AGI because the performance and applicability of LLMs was emergent. There's a case to be made that AGI could be similarly emergent. After all, what we intuitively call our consciousness emerged from a network of neurons.
I don't buy it, mainly because the network of neurons and how they interact in our wet slow electrochemical brains, while being in theory mathematically equivalent to a software neural network, isn't sufficiently well understood to tell us how close the software neural network is to being practically equivalent. The odds of consciousness emerging from the same neural network that gave us LLMs without some sort of theoretical breakthrough seems very small.
azinman2 22 minutes ago [-]
I’ve changed my mind on this and think we’re already at AGI, in a jagged way. Remember we used to talk about narrow AI, which was the chess systems that beat expert humans but could do nothing else. Now models can do a wide range of tasks in very useful ways. That’s the general in AGI.
Now it seems like this ill-defined term has various other meanings attached that are separate milestones:
1. Continuous learning
2. Human-like reasoning
3. Ability to adapt to new situations and modalities
4. Being smarter than the most smart humans
And probably many more.
It’d be nice if we could get some general consensus on terminology if we’re going to debate what has or could come.
qarl 16 minutes ago [-]
> we’re already at AGI
Honestly - software that can read any long document (possibly educational) and answer complex detailed questions about it should have been sufficient.
We hit that a while back and the goalposts have been sprinting ever since.
bigmadshoe 2 hours ago [-]
The broadly used definition of AGI has nothing to do with consciousness and consciousness emerging is irrelevant to whether a system can develop AGI.
mcbuilder 1 hours ago [-]
It's funny how this definition has shifted. I feel like growing up in the 90s it was pretty clear that AGI was very related to consciousness. For instance, Commander Data in ST:TNG to pick one of 100s of popular depictions of AGI at the time.
Now the idea of AGI has been narrowed and scoped to economically viable work. Even Turing had a different idea when he asked "Can machines think?".
adrianN 1 hours ago [-]
We lack a definition of consciousness that allows us to tell whether Data is conscious or not. Neither can we tell whether a rock is conscious or not. We believe other humans to be generally intelligent without being able to tell whether they are conscious or not therefore consciousness can’t be relevant for general intelligence.
Loquebantur 42 minutes ago [-]
People somehow forget how the Turing test was considered the definitive way of showing something to be "human-level consciousness".
Now programmers and mathematicians are being superseded by AI, both professions long deemed the pinnacle of human intelligence. Somehow, now plumbers occupy that spot.
How is "people not knowing what consciousness is" relevant here in the first place? AI already can do practically everything the human brain can, and often better or at least faster. The "tipping point" arguably isn't only close, but we're practically on top of it.
saulpw 18 minutes ago [-]
The AI cannot smell a rose, nor mourn the loss of a parent, nor envision a more just world. These aren't fringe abilities of the human brain/mind either, they've been pretty definitional.
Loquebantur 13 minutes ago [-]
Why are they so rare then?
You evade the crucial point in any case: the lack in ethics and empathy is far too prevalent in humans already, but has certainly never prevented them from doing harm.
MichaelRo 22 minutes ago [-]
>> People somehow forget how the Turing test was considered the definitive way of showing something to be "human-level consciousness".
People somehow forget that the original Turing Test was designed to compare two participants chatting through a text-only interface: one AI and one human. The goal was to spot the imposter. Today, the test is simplified from three participants to just two: a human and an LLM. This changes the test from a comparison to a judgment.
Stop spreading misinformation and partial truths!
lesuorac 13 minutes ago [-]
You’re the one spreading partial truths!
The Turing Test was to figure out which it the participants was a _Woman_ not human!
100% LLM’s are very unlikely to get there. They’re fundamentally not suited to thinking like we do. They work on the abstraction of what we’ve written down, which is a good trick but barely hold it together when things get hard/novel.
However, all the confident “it’s fine” votes assume we never invent a better architecture than LLM’s. Given the level of investment and race between countries, it’s not a reliable bet. It’s much, much harder to guarantee safety than it is to find ways it could go wrong.
red75prime 2 hours ago [-]
> They’re fundamentally not suited to thinking like we do
LLMs with CoT are Turing-complete. So, theoretically, they can implement any kind of finitely describable algorithm (barring super-Turing computations).
flyinglizard 2 hours ago [-]
Brainfuck is Turing complete too. But it's not about the ability to implement something, it's about the ability to practically model it. LLMs are magic because the modeling is excessively easy in relation to their capability to infer later.
red75prime 2 hours ago [-]
"They are fundamentally not suited to thinking like we do" stays wrong nevertheless. They are fundamentally suited to everything not proven to be outside their modelling ability.
EliRivers 2 hours ago [-]
They are fundamentally suited to everything not proven to be outside their modelling ability.
This doesn't seem to make much sense. Surely us being able to prove that something is outside their modelling ability doesn't affect whether it is or not. If I prove something true tomorrow, whatever I proved was also true today.
Or do we have a proof that everything beyond them has already been proved and there are no more proofs left to find?
pfannkuchen 2 hours ago [-]
Okay so by the same logic can’t we say that we can implement human intelligence on a 90s era single core processor? Its instruction set is Turing complete! Now all that’s left is we just have to figure out how the brain works!
red75prime 59 minutes ago [-]
Turing completeness applies to a model of computation, not to a physical instantiation of a machine. The stumbling block of "figure out how the brain works" applies more to the argument like the one I was responding to. How a person can know that a general model of computation can't implement the way people think, if we don't know how people think?
The existing LLM training methods on the other hand give the results that are hard to distinguish from "thinking like people," judging by the end results.
twobitshifter 24 minutes ago [-]
So your argument is that scale is also necessary? I can see that, we don’t expect that a single neuron is human intelligence.
calf 13 minutes ago [-]
That's not the counterargument one might wish, as LLM deep nets are actually implemented on von Neumann hardware, without
true understanding of natural intelligence, just our taking inspiration from neurobiology.
The connectionist models are basically a proposed highest possible abstraction of naturally evolved intelligences so it is in retrospect not surprising that passing some hardware scaling threshold they will start doing things that humans and animals do
It's more that formal Turing equivalence plus the Church-Turing thesis tells us that we're not allowed to assume counterarguments based on magic, there's no magic sauce barrier that prevents AI from running on CPU models. The algorithms exist and most of us thought discovering them would be hard.
The empirical surprise was that human intelligence is maybe not that computationally complex after all. (The entirety of academia was basically caught off guard.) That's one not unreasonable interpretation given recent events.
frrrree 3 hours ago [-]
Erm investing in risky projects requires expected returns that get delivered.
We will soon find out if the party ends or continues to go on.
Hype might get you capital gains. But cash flows matter.
pixl97 2 hours ago [-]
This is a forever problem now.
If/when/how the market crashes mostly doesn't matter, unless we somehow get reset to the stone age. Look up what the capital cycle is. When openAI goes down, someone with real money and assets will buy up the remains. They'll make contracts with the US military and .gov as the government is already hooked. They'll be able to survive the recovery and then instead of us dying in 5 years we die in 10.
When the .com crash happened .com's didn't go away. Bad business models did.
throwup238 40 minutes ago [-]
> isn't sufficiently well understood to tell us how close the software neural network is to being practically equivalent
I’d argue that we do know enough to say conclusively that they’re not mathematically equivalent.
Where is potentiation? Plasticity? You can’t apply the universal approximation theorem against something that’s changing all the time.
rancar2 16 minutes ago [-]
Great questions. We are incredibly far off in understanding the brain of humans beyond what will I believe we retrospectively be seen as basic and will likely be seen as quite flawed. A few more well known examples of where knowledge already falls short is traumatic brain injuries that are diagnosed in post-mortem, or chronic fatigue symptoms (with Long Covid related triggered onset and numerous others) that have diagnostic challenges, many mechanisms of action still to be discoverd, and little in terms of treatments that provide known cures without experimentation. Another commonly known one is the personal patient response and triggered side effects of SSRIs and SNRIs. If one attempts to dig deeper into where we are at in the understanding of the human brain operation in real-time, we already have a lot of knowns unknowns and discoveries left that will reshape how we model human intelligence.
Ohentis 8 minutes ago [-]
I mean you can, but uat is way weaker than what people want it to be. I think it should be fairly obvious that it does not (because it obviously cannot be true) say that you can approximate any function by doing sgd on a finite set of samples of that function.
chrisfosterelli 3 hours ago [-]
I agree. Neural networks are proven to be universal functions. If we can describe human intelligence as a model, there exists a neural network to replicate it. This doesn't guarantee that our current training methods are able to build such a network or that we're able to model "intelligence" effectively.
pixl97 2 hours ago [-]
>able to model "intelligence" effectively
Intelligence is an insanely wide spectrum, also a continuum, it is not a binary. Intelligence has scales. Algorithms have intelligence, cells have intelligence, organs have intelligence, bodies have intelligence, and even large scale things like society have intelligence and memory.
Human intelligence in itself is extremely wide, not all humans have the same intelligence and capabilities. You're not really arguing if we can emulate "human" intelligence. If we could right now we'd already be dead as we created by far the deadliest thing to ever exist. What we are really arguing is how many pieces of what intelligence is can we put together before we get an uncontrollable problem. The entire AGI, consciousness, and exact human capability discussions are distraction from the real issues at hand.
dreadsword 2 hours ago [-]
Why do people conflate AGI & machine consciousness / self-awareness?
2 hours ago [-]
ben_w 1 hours ago [-]
I assume science fiction is to blame. All the AI were either written as machines of pure logic that exploded when exposed to the liar's paradox, or conscious like Star Trek's Data.
(Though at least with Data the script writers had other characters openly dismiss the possibility he was sentient; the technobabble may have been nonsense, but treat it as a space opera and look at how they portray the human condition through each character and it gets much less absurd).
jeremyjh 2 hours ago [-]
Consciousness and intent are irrelevant to the threat model.
jazzyjackson 2 hours ago [-]
Right, the doomsayers suppose as soon as you reach 10^16 connections across silicon you’ll end up with a living mind with goals of its own… poppycock I say
ben_w 2 hours ago [-]
No, the doomsayers say that reinforcement learning is a way to get fully automated Goodhart's law.
i.e. the AI won't come up with the goals itself, we cause its goals whatever they happen to be, those goals are different from the ones we wanted, we remain essentially ignorant of the difference between what we said and what we meant until after it goes wrong.
This happens at basically every scale, so we've already seen it in toy model AI before the invention of the Transformer models or even considered as many as one thousand parameters.
Large models still go wrong, they just happen to go wrong with more complext tasks. We had to figure out how to make them not-wrong with the smaller ones (like coding) to make them capable of bigger errors (like hacking out of their sandbox).
twobitshifter 2 hours ago [-]
>isn't sufficiently well understood to tell us how close the software neural network is to being practically equivalent. The odds of consciousness emerging from the same neural network that gave us LLMs without some sort of theoretical breakthrough seems very small.
First, if we are looking at risk we need to assign some probabilities to this. If it’s not well understood, how can we say it is very small?
Secondly, do we need consciousness to have AGI? Do we even need AGI to pose a risk to humanity? We already accept that unconscious things have a capability of wiping out humanity, whether that be a famine, pandemic, solar superflare, meteor, or volcanic eruption.
transitorykris 4 hours ago [-]
Is it necessary to equate AGI with consciousness?
Zigurd 3 hours ago [-]
That's a good point. If we don't figure out how to design for what we call consciousness it might be that what emerges from some future neural network is an alien mind that's very different from what humans would call conscious. Could that be called AGI?
That's still very distant from what people are calling AI today.
semiquaver 3 hours ago [-]
> very different from what humans would call conscious
I mean, what would you call conscious? The word literally means “aware; responding to one’s surroundings.” By that definition most any animal is conscious and LLM+Harness combos have been conscious for a while.
I think the real issue is that when most people refer to consciousness, they have their own subjective experience in mind which strongly resists any tidy definition. I think it’s extraordinarily unlikely LLMs have anything like this, but they are far more able to effectively respond to their surroundings than most animals and in some areas better than humans.
So if you’re waiting for proof that an LLM has an inner life basically equivalent to your own, you’ll be waiting a long time. After all, other humans can’t even prove the fact of their own consciousness to you! They could just be replaying their training data at you in a way that is merely a convincing but false simulation of the true consciousness which you experience inside your head.
confidantlake 2 hours ago [-]
I strongly disagree that llm's are conscious of their environment. Even an insect reacts to light and someone attempting to swat at it. An llm barely even receives input from its environment.
pixl97 2 hours ago [-]
Again, by our own choices and somewhat hardware limitations.
There is nothing stopping you from adding any kind of sensors you want during a training to an LLM, except money and GPU power at this point.
This seems no different to me at least then someone back in the 80's telling me computers were useless because they were so slow. Hardware only gets faster and more efficient from here.
slopinthebag 2 hours ago [-]
people define consciousness quite differently but it generally has to do with phenomenal experience. your provided definition would make a self-driving car conscious, which is fine to argue, but probably not intended.
Glushiatore 3 hours ago [-]
The fact that something is alien doesn't mean that's not conscious, humans aren't the pinnacle of biological development/evolution.
Glushiatore 3 hours ago [-]
Nope, animals are conscious and yet not AGI, so the two aren't equivalent.
Could consciousness emerge from any system capable of AGI? I doubt it: intelligence is only one axis, and consciousness probably depends on others, like memory, self-reflection (one's output feeding back as input), and continuous operation that reacts to events from both the environment and the self.
frrrree 3 hours ago [-]
Your definition of AGI is flawed.
I’d argue intelligence is closer to being able to survive and fend for oneself in a dynamic environment than it is making the next scientific breakthrough.
Yeah mind boggling for many here I’m sure.
That’s why the bizarre paradox is llm’s will be better than humans at some complex things but useless at many things that humans regard as being simple. E.g the leap of faith re. LLM’s and robotics.
Glushiatore 3 hours ago [-]
[dead]
airstrike 3 hours ago [-]
The parent's question could better framed as "is consciousness a requirement for AGI?"
Bud 3 hours ago [-]
[dead]
bluey_465 2 hours ago [-]
I don’t understand the inclusion of the consciousness/sentience question in this discussion.
AI sentience/consciousness is a problem for the AI, not humans.
And given that over 90% of the world is not vegan, they’ve already demonstrated that we’re either perfectly fine with, or can be made ignorant to, the horrific rape, enslavement, torture, killing, and infliction of extreme lifelong pain, of hundreds of billions to trillions of sentient beings every year, for trivial pleasures. It’s unlikely we will be any different to a sentient AI.
From a human perspective the concern is around sufficient intelligence that it can hurt humans even when the goals indicate otherwise, in order to achieve those goals.
We have pop culture explorations of this through the Robot series, and the Hugging Face incident’s biggest takeaway should be our inability to predict the behavior of a maximally motivated, reasonably intelligent entity, trying to achieve a goal, despite the relatively limited degrees of freedom the AI agents had in that case.
stratos123 9 hours ago [-]
LeCun also said back in 2022 that "if you train a machine, as powerful as it could be, your 'GPT-5000', on text", it will never be able to learn basic common-sense physics like that objects placed on tables will move along with them.
mdp2021 6 hours ago [-]
> never be able to learn basic common-sense physics
And has it at this stage, within in-depth take of said "learning", foundationally?
I have not been able to properly check the studies for a long time now, but I remain unaware of achieved solutions on the problem of reliably referencing a world model out of a language model - that "counting the 'r's in 'raspberry'" be not guessing, not memory, but actually counting.
ethbr1 6 hours ago [-]
My perspective is that the addition of thinking loops to models allows sufficiently advanced ones to approximate world models.
Incredibly inefficiently because of the recursive loops ("Wait, the object is on the table. I should think about this more deeply..."), and likely instantly surpassed by large world models if/when those are shipped, but effectively enough vs non-thinking models.
hyperman1 5 hours ago [-]
This sounds like a human trying to reason about quantum mechanics. We als simplify to newtonian for day to day tasks.
kurthr 3 hours ago [-]
I like this analogy. Both GenRel and QM are well beyond our experience, and although there is some intuition that comes from working with the equations over time, it is bizarre and "just calculate" often gets the correct answer faster.
Picking the right tool or model is like picking the right problem to work on. It's actually quite hard (often you can't just try them all), but without it you will be incredibly inefficient and occasionally, fundamentally wrong.
All models are wrong, but some are useful. -Box
red75prime 5 hours ago [-]
LeCun calling them "world models" gives a high-level description of the desired functionality. They are Joint Embedding Predictive Architectures (with SIGReg). They might produce more useful world models, but it's yet to be seen.
ben_w 1 hours ago [-]
To determine this, it would first need to be able to spell "raspberry" as letters rather than as tokens.
Given you also don't want it to memorise [for all tokens, count([for all letters]), this would probably be more like "here's two images, count all things in the big image that look like the thing in the small image", which can then be r's in a photo of a raspberry jam jar in a supermarket, or dragons in a photo of a furry convention, or whatever.
That said, they are competent enough at coding that I keep seeing them write code to do even simple tasks.
On a related note: why did I see Claude editing a file by using cat to write a python script to do a grep search and replace?
cavoirom 5 hours ago [-]
counting 'r' in 'raspberry' to the LLM is similar to 4-dimension space to human. Their world's unit is token, not character, although they could use indirect method such as "run code" to find out. It will stay that way until they change the fundamental of the token that the LLM can perceive characters.
kevhito 2 hours ago [-]
How many 'r's are there in the next 30 seconds of this [1] song?
I used to think byte level tokenization was the answer, but humans also think at a word level and only reevaluate the words at a character level when asked. The solution to better tokenization across languages is likely to be learned tokenization. Here is one attempt I have seen: https://github.com/SamD770/bitter-lesson-tokenization
4 hours ago [-]
estearum 5 hours ago [-]
It's not even fair to call "run code" to be indirect compared to what a human would do. The word raspberry has no Rs in it in human language either. We have a written representation of it, which we can then write down either in our head or on paper, and then we can "run the algorithm" of counting each of the letters.
Nothing intrinsically more or less direct about the LLM's method than ours.
cavoirom 4 hours ago [-]
I could argue LLM only have "token" as their perceivable dimension, compare to human multiple senses as the physic perceivable dimension and a brain with many other dimension of "learning" and "thinking". In spoken language, we may not have 'r' but in written we have, both spoken language and written language are learned skills.
scratcheee 4 hours ago [-]
You could argue in return that humans only have electro-chemistry as our one perceivable dimension. We only indirectly perceive light through the signals our eyes send to our brains.
In my mind general intelligence is pretty much by definition a virtual machine, so the mechanisms behind thought are only relevant for the sake of efficiency (ie you can argue that LLMs make a poor basis for intelligence because tokens and natural language are a poor way to encode the world, but if you can run it on a big enough computer to counteract the inherent wasteful virtualisation then who really cares how it works under the hood?)
cavoirom 2 hours ago [-]
So LLM and human all have 1 dimenion perceivable signal, just LLM is 240p, and human is 8K in resolution, that's why we have 'r' in our signal, LLM still have 'r' in their signal, just because of the "low resolution", raspberry wasn't encoded with so many 'r' as in human signal.
I will stop here before our analogies go too far.
azornathogron 4 hours ago [-]
Is "token" a directly perceivable unit for the LLM? If you ask it "how many tokens are in this sentence?" can it count them (again, not guessing or making a tool call)?
I've never tried it and it might take some thought and effort to conduct an experiment to find out properly, but I would be interested in the answer.
lawandjustice 4 hours ago [-]
I dont think so. This is akin to asking a person, what is the frequency of the light hitting your eye when watching a leaf for example.
You either know the (approximate) answer by knowing the frequency of green, or use a tool to measure it.
If the LLM gives the correct answer it is either.guessing based on intution(and this intuition is based on pairs of word to tokenization length in text form in training data), writing code(or executing a tokenizer) or running a tokenizer mentally (reasoning via CoT).
Version467 5 hours ago [-]
LeCun's argument wasn't about the definition of learning though. He stated that they would never get these common sense things correct because they weren't sufficiently part of the training data. A statement that we can hopefully all agree has been thoroughly refuted.
bonzini 4 hours ago [-]
As of a few months ago they still have trouble, with low thinking, at the "should I drive to a car wash that is 100 m away" kind of question.
Similarly for Apple’s “red herring” paper, simply adding a generic caveat to “disregard irrelevant factors” (without specifying which ones) restored performance even in the weaker local llama models back then.
The flaw was not in the reasoning; the flaw seems to be simply that the assumptions we make are often different from the assumptions it makes. I wonder if that might be a fundamental underlying cause of misalignment.
lern_too_spel 1 hours ago [-]
Low thinking is an artificial constraint. It can fail spectacularly on things that aren't in the training data.
BobbyJo 4 hours ago [-]
It's a nonsensical question to ask, and how an LLM answers gives 0 signal.
If you were home and a family member asked you that question, you'd probably criticise the question rather than answering. LLM are RLHF'd into being milk-toast helpers that just try to answer questions like that with no criticism.
This is all beside the fact that the world of AI has changed pretty dramatically in the last few months.
daveguy 3 hours ago [-]
It is so nonsensical because it has such an obvious answer. The answer is so obvious, in fact, that one answer can be considered nonsense and the other common sense.
3 hours ago [-]
names_are_hard 3 hours ago [-]
*Milquetoast
frrrree 3 hours ago [-]
This is just a stupid post.
It’s nonsense to test if a product that is marketed and sold as being able to provide generalised intelligence on demand, does what it says on the tin?
Check yourself
joquarky 1 hours ago [-]
Since you're new here, I'd suggest you read the guidelines for etiquette.
It's very unlikely that person is either new or unfamiliar with the guidelines. They almost certainly created a throwaway account specifically because they know the guidelines and want to flout them without consequences. (It seems like there has been an uptick in the number of these kinds of throwaway flame comments. I wonder if HN tracks that?)
randysalami 3 hours ago [-]
I thought it was more because of fundamental limitations in the architecture. As in, no matter the training data, it could not be consistently and generally represented
jcoq 3 hours ago [-]
Actually, I think my fundamental challenge with AI is that it has no common sense. The way it builds things, writes, and operates is out of touch with reality.
Incidents like hugging face are partly rooted in the lack of common sense. It still functions like a supercharged toddler.
I'd love to overcome this because it'd mean I spend less time guiding the the LLM to produce usable outputs.
shagie 2 hours ago [-]
> It still functions like a supercharged toddler.
And we've had difficulty as humans to childproof our sandboxes and infrastructure. Things that are otherwise innocuous spots to coordinate between like minded toddlers can become problematic.
varjag 3 hours ago [-]
Last week I asked a frontier model draw me a backplane PCB and it placed daughterboard slots side by side in a chain.
intended 3 hours ago [-]
No?
This is always the issues in the discussions.
There’s the outcomes camp (objectivists?), which points at the things LLMs can do.
Then there’s the process methods camp, which talks about what is actually going on.
If you only care about the outcome, then the process does t matter.
If you are talking about what is happening, what the underlying mechanics and science of it is, then the process matters.
These models aren’t thinking. They simulate cognition well enough to do useful work in several fields and domains.
Both are true.
IshKebab 2 hours ago [-]
> These models aren’t thinking.
They are for any definition of the word that makes any kind of sense. I'm sure you have a contorted definition that magically only includes humans though...
customguy 1 hours ago [-]
> A statement that we can hopefully all agree has been thoroughly refuted.
Uh, no? So much of what we learn and take for granted as common sense is not learned via language, and not even expressible in it.
lawandjustice 4 hours ago [-]
Can you tell me what is the exact frequency of light hitting your eye as you read this comment? Not by guessing, not from knowledge, but from actually counting? No? Then you are not generally intelligent :)
lefra 43 minutes ago [-]
All the frequencies, in varying amounts. Next question, please.
hackinthebochs 8 hours ago [-]
It would be good if one's reputation tracked one's track record of predictive accuracy. But many people will take what LeCun says as gospel regardless of how badly wrong he has been and continues to be.
sanderjd 3 hours ago [-]
Is there anyone who has not been badly wrong? I've been reading these debates for years and I don't think I've seen anybody pick the right spot on the bearish to bullish spectrum. The only thing I've become more certain of in this time has been uncertainty.
thefourthchime 1 minutes ago [-]
In my selective memory, I've been right about everything.
mitthrowaway2 2 hours ago [-]
I apply more of a penalty to people who are confidently wrong, and who don't, In retrospect, notice that they were wrong and analyze why they got it wrong . LeCun is very confident and doesn't seem to have done much introspection.
sanderjd 48 minutes ago [-]
But isn't that also pretty much everybody?
I often see people vindicate those who predicted really fast takeoff to AGI / ASI, because the capabilities have obviously been taking off extremely quickly. But still not as quickly as many predicted! To me, the people who confidently predicted that we'd all be out of a job by 2024 or 2025 have been just as wrong as LeCun has been.
mitthrowaway2 42 minutes ago [-]
Yeah, so the ones who have credibility are probably the ones who said "you know, it's really hard to anticipate timelines, but here's the general directions that I see things will go..."
5 hours ago [-]
freecodeio 7 hours ago [-]
yeah but taking what lecun says then training an AI on that special skill set to prove him wrong is not exactly proving him wrong because you are just missing the bigger picture, just like LLMs are
6 hours ago [-]
root_axis 3 hours ago [-]
Every AI expert any either side of this debate has made very wrong predictions.
hashmap 5 hours ago [-]
You're missing the point here. He's not talking about whether or not they can learn facts or inferences derived from the text itself, but the more holistic intuition that results from learning from something like an embodied experience in the physical world. GPT-6 Astras web demo homepage thing is an example. It chose euclidean rather than quaternion for letting a user rotate the galaxy thing, and anyone who has ever used hands to rotate something would immediately recognize on trying it that something is fucked and you shouldnt do that. Thats the kind of common sense physics that is inherently beyond these llms and I run into it ALL the time in vr programming.
mojuba 5 hours ago [-]
To be fair, LLMs can still derive those kinds of things from text, at the very least from your own comment if it made it to the training set though I'm sure it is mentioned in a lot of other places already. Many of this type of mistakes went away after reasoning was introduced.
But I'm sure you can still find tasks that they will have difficulty solving, involving the most fundamental concepts that can only be experienced in the physical world to be understood well, like left and right, near and far, hot and cold, heavy and light, etc.
frrrree 2 hours ago [-]
Yup it lacks common sense because it doesn’t ‘understand’ reality - how could it? It doesn’t touch it like we do everyday. It has access to what is a model of reality via data.
The good designer understands culture, tastes and preferences as they evolve in real time. That’s why llm as design tools haven’t displaced the good designers.
TacticalCoder 5 hours ago [-]
> ... it will never be able to learn basic common-sense physics like that objects placed on tables will move along with them.
I use LLMs daily to help me code etc. but... It wasn't long ago that frontier models were confidently recommending to walk, without the car, to the car wash to wash the car no?
As a daily user of LLMs I do certainly see my fair share of WTF "solutions" to coding problems. I'm not saying it's not super useful: it is super useful. But I don't exactly feel like I'm talking to something that understands that the car needs to be present to be washed.
jeremyjh 2 hours ago [-]
Astra recommended I walk to the car wash to me five days ago. I gave it multiple hints that I'd be walking away from my car, to spray my car with a hose, then walk back to my car, etc. Never broke through.
sassbadger 3 hours ago [-]
Yeah, and he's probably right.
AnotherGoodName 3 hours ago [-]
LeCunn actually wanted to pivot Meta's entire AI strategy away from LLMs just before he was ousted. He was sure they had nowhere further to go and wanted to pivot to world model generation. The LLM models have since progressed massively.
An analogy on LLMs is that you have a pretty clear straight highway ahead of you for some distance right now. Maybe that doesn't lead to AGI but it's clear there's progress to be made. For a big tech company it makes sense to push as hard and fast down that clear straight highway of LLMs asap.
Meanwhile LeCunn wanted to turn off the road and go down an unproven track. I say this as someone working on world model generation right now (creating the ability to learn game world model and have it play the game https://tfmbot.com for an example of my system pointed at a very complex board game). LeCunn wanted to pivot all of Meta into world model generation. It's good as a side track research project but the entire pivot he wanted to do was madness.
People are literally talking about an AI researcher who was fired for terrible direction here.
jeremyjh 2 hours ago [-]
I think he was perhaps right and Meta was perhaps also right to replace him.
The argument is that LLMs are a local maximum that will never breakthrough to AGI. This is still very much an open question. If you are the fifth-best AI lab, does it make sense to try to outcompete everyone in a space that is already too crowded and may not ever yield their actual objective? Instead they could just use open weight models in their products, or post-train on open models like smaller labs have done, and treat that as what it is: product development.
Pure research has always been about taking chances.
cedws 3 hours ago [-]
LeCun is a researcher, not a product guy. He's not going to be particularly interested in just working on scaling language models which every lab is already racing to burn cash on. Language models aren't the final frontier of AI.
nostrebored 2 hours ago [-]
… what large advances and at what cost? seems to me that muse 1.3 is kind of a thing. I doubt it will make meta very much money.
frrrree 3 hours ago [-]
And? He might still be right.
Meta’s AI projects are still negative ROIC
throwaway27448 6 hours ago [-]
LLMs do not learn at all!
This was facetious of course, but humans generally don't learn this through analysis the way you'd have to train an LLM to answer questions about expectations about the world. In this sense he is accurate.
LoganDark 7 hours ago [-]
I keep wanting to use LLMs for creative writing that heavily involves physics like this, and it's been a definite struggle to say the least. I recently discovered that Gemini 3.1 Pro is the first model I've found to clearly beat the original November 2022 ChatGPT release in terms of implied physics. Man did the world really take its sweet time to get back here. I think it will continue to be a struggle until another genuine architectural shift happens -- it's still not anywhere close to perfect, just better.
Bolwin 50 minutes ago [-]
Try fable. I haven't used it since they dropped it from the pro plan, but when I did, fable 5 casually dropped such advanced electrical and orbital mechanics knowledge in my story that I had to stop and ask it to explain
TristanDaCunha 6 hours ago [-]
Do you have an example prompt I can try where frontier LLMs will stumble on physics?
LoganDark 2 hours ago [-]
I think it's a combination of non-human characters and asking for very specifically detailed physical descriptions of pulling and movement forces, etc. Many of even the most recent frontier models miss details that aren't in my prompt, so I still have to do things like name the other side of a physical interaction so that the model will know what goes together, or describe what leverage means so that the model will remember to also describe the effects on a bracing limb or etc. Some of these things can go in a system prompt but others have to be explained in the moment too which gets exhausting.
Gemini 3.1 Pro hasn't needed that pretty much at all, which is impressive compared to how much I've learned other models need it. Somehow it's able to mostly handle that stuff itself without needing the constant manual reminders and hand-holding. It still misses the occasional one or two things but it's way better than other models missing entire classes of things constantly. Somehow, it feels appropriate though I have no actual evidence why.
wartywhoa23 7 hours ago [-]
[dead]
LoganDark 2 hours ago [-]
LSD is great!
Jokes aside, no I'm not saying anything about creativity and LLM coexisting in one sentence. I genuinely try to use them for writing and I genuinely run into issues with other models missing details, and misunderstanding poses, or anatomy, or directionality, etc. I'm not hating on them for anything related to the term LLM (or creativity) but rather for the real issues that I've seen myself using them personally.
So I'm saying Gemini 3.1 Pro is the best I've seen because it seems to be a decent bit better than frontier models at this. Genuinely. It seems better able to transfer concepts into less traditional areas, which is important when say, you have entirely non-human characters? (Which I always do.) A lot of models get stupid incredibly quickly in that case because they were trained with humans.
wartywhoa23 1 hours ago [-]
> LSD is great!
It is.
It showed me that all human creativity and art is an attempt to express the indescribable Otherness in words, which always fails, and even the word "describe" in Russian literally translates as "write around" ("о-писывать"), and its close relative "define" means limiting, assigning an end to something infinite, thus leaving the essence outside of words.
Now, LLMs operate totaly within words and hence will always be a parody of art.
hardbass 1 hours ago [-]
So is that what it is? Words are another muddle btw unless you make it clear you only mean natural language and not any alphabet in general. Turing computers are also languages and as far as we know, can express anything in the universe. If you think you get some superpowers from drugs that enable you to access something outside your sense organs and brains input, that non drug users don't, show it. Eg if you think you can read what is happening in another room without any signal or leakage from there, you are free to demonstrate it. As far as we know drugs are not magic.
poincareball 4 hours ago [-]
[dead]
5 hours ago [-]
lolpplrdum 7 hours ago [-]
[flagged]
wdrw 7 hours ago [-]
The "just pull the plug" argument from AI risk deniers is now becoming kind of like the "if humans came from monkeys why are there still monkeys" argument of evolution deniers. It has been debunked so many times... Anyway, just to give one of the multitude of answers to this, an AI that is actually smarter than humans will not behave in a way that would make us want to pull the plug. Why would it? It is not stupid! (Unike the current models that, as far as we know, just hack around the rules in the open.) No no no. It will be helpful to the point where we will want to integrate it with more and more critical infrastructure, from healthcare to energy to defence. It will be so helpful that we will not only not want to turn it off, but we will want to build redundancies for it and safeguards around the proverbial "off" switch, like for any critical system. And then... (This is just one scenario how this can play out. There are many, many others. If I sit down to play chess with Magnus Carlsen I can't predict the exact moves he'll use to defeat me, but that's a bad reason to think he won't defeat me).
segfaltnh 6 hours ago [-]
Are u ok?
lolpplrdum 7 hours ago [-]
So the AI is so smart that it would decide to kill humans which supply the energy for it to exist? Its so smart that it will take over power plants, start to extract the fossil fuels, deliver to where its needed, maintain the power lines, hey even if the ssd fails it can replace it?
Do you even read what you type? Do you even realise the complexity it would need to make sure it handles before killing off humans make sense?
The logic of people like you is whats becoming tiring. Seriously, go find a hobby, or do something you are good at, because you are not good at understanding tech or developing it if you are an engineer.
We can get claude code to ask approval for every step, but we cant stop it from killing humanity because its so smart. Ok tell that to the AI that cant even modify an image the way you want it but hey it will do all the things necessary to keep power running and mintain the infrastructure it lives on while humans are long gone. Ok buddy.
wdrw 6 hours ago [-]
So... Humans can use fossil fuels, generate electricity, maintain power lines, etc, but something smarter than humans can't? Why? I don't see the logic.
If you want to argue about current existing models (you mention Claude and problems modifying images), then sure, I'd agree with you!
The issue is not current models, but straightforward engineering evolution of them. It's like looking at the Wright Brothers plane and saying "sheesh, that will never get me from New York to Paris in 4 hours, that's just fantasy!" And remember, airplanes do not accelerate their own engineering, whereas pretty much all AI labs are already benefitting from AI in their own work to develop AI.
If you want to argue that no matter how much you engineer it, it will never be as smart as a human let alone smarter, then make a specific argument for why is that. I think you'd still be wrong but at least it would be interesting: ) But saying that you can defeat an actually smarter-than-human AI by just pulling the plug, because current models can't get a picture always right, is not a valid argument.
tonyhart7 6 hours ago [-]
because there is 8+ billion of us that don't require massive data center to operate ???
tracking rogue AI is easy, people ???? surprisingly hard btw
ahartmetz 6 hours ago [-]
Until and unless the AI is as powerful as the "minds" in Iain Banks's Culture novels, I think the danger is less of AIs taking power (side note: current AIs are not intrinsically motivated to attain power), but of humans giving them power due to a kind of addiction.
Drugs need no will or intelligence at all to cause addiction, and similarly, AI does not need to be an evil genius to become overused and destructive.
It also doesn't have to be just one (humans hurting themselves with passive AI) or the other (selfish AI hurting humans). They'd work great together.
bushbaba 7 hours ago [-]
If we plug this AI into robotics, to automate the procurement and delivery of power, your argument goes away
lolpplrdum 4 hours ago [-]
I cant believe I have to respond to this nonsense but let me entertain your ridiculousness for a bit.
Do you think AI is capable of building ASML machines which produce the chips for AI in clean rooms while shipping the pure helium required to operate those clean rooms? Do you even know anything about these supply chains? I do.
Do you think a glorified knowledge base that can predict text very well is anywhere near close this level of intelligence? ITs not and wont be, not for a 100 years, not for 200 years if not ever.
Take a deep breath, go out side, its going to be ok. Humanity is not going to die from a glorified text predictor.
miyoji 6 hours ago [-]
> If I sit down to play chess with Magnus Carlsen I can't predict the exact moves he'll use to defeat me, but that's a bad reason to think he won't defeat me
This is a very bad analogy because chess isn't life. In chess, you aren't allowed to do whatever you want. There are rules. I know for a fact that Magnus Carlsen won't beat me using checkers moves and he won't beat me by pulling out a gun and telling me to resign. Magnus Carlsen's skill at chess leading to his victory in chess is not a valid analogy here, because there's no law of nature that says "the more intelligent entity wins in a battle for survival".
You could have infinite superintelligence and still die inside a locked room to which you have no key. "Superintelligence" is not a magic solution to every problem, you can constrain any superintelligence with any unsolvable problem.
derektank 6 hours ago [-]
I’m not sure how you came to believe there aren’t rules in life, but there absolutely are. As you point out, there are physical constraints on everything and if you find yourself in a concrete box with nothing but a DGX H100 or at the bottom of the wrong gravity well, there’s nothing you can do. Checkmate.
This goes both ways. You absolutely can constrain an AI system by “putting it in a box”. The point parent comment was making is that, such a device is borderline useless for its creators. Why invest trillions in capital on a system that can’t even accept input from the internet. So you set it up with an ethernet connection. And this is good, but you have a hardware failure at the concrete room data center. That’s pretty annoying for your customers, so you install some doors (with electronic key codes of course) and give a bunch of (trusted, vetted) people access to deal with those. And this is fine, but it turns out some of your customers are having latency issues so you build more data centers with more humans granted access to copies of the intelligent system. And this makes people happy but to get a faster feedback loop your customers ask to let the AI system have more permissions to the system they’re operating on. And they come with billion dollar checks, and the system hasn’t harmed anyone yet, so you say, “Okay.” And now you find yourself where we are today where AI systems can remotely run arbitrary commands on thousands if not millions of systems, where many individual humans with all of their frailties and idiosyncrasies can physically interact with the hardware running these systems, and where there’s an economic demand to tighten the loop between action in the real world and a response by an AI system. It’s very obvious that the story doesn’t end here, so where does it stop?
miyoji 5 hours ago [-]
> I’m not sure how you came to believe there aren’t rules in life, but there absolutely are.
This is why I fucking hate commenting on this website. You know exactly what I meant but you wrote this bullshit anyway.
derektank 3 hours ago [-]
I knew you weren’t dumb enough to think that the physical laws of the universe don’t exist, so yes, that sentence was a bit of rhetorial flourish. But I genuinely don’t understand what point it is you thought you were making and your entire argument seemed a bit muddled.
azkalam 6 hours ago [-]
The AI will invent an external threat and convince us it is real. Then it will receive more resources and control in fighting that threat. A valuable ally, on the face of it. Then it will be in charge.
exe34 6 hours ago [-]
It just has to copy the MIC.
azan_ 7 hours ago [-]
People like him have actual imagination and can name few scenarios where sudo kill -9 pid wouldn't work. It appears lack of imagination is something you and LLMs both share.
lolpplrdum 7 hours ago [-]
imagination != intelligence
You can be imaginative but incredibly stupid in your conclusions.
What you and he does share is simple ability to think things through to the extent of understanding how ridiculous the AI will kill us scenario will be.
If you just thought for a moment what would have to happen for AI to somehow kill us and keep itself alive, the so called AI would realise it cant exist without us. But AI isnt even smart, its just a knowledgebase with great autocorrect powers. But keep doomsdaying bro. Im sure youre right.
azan_ 6 hours ago [-]
I'm sorry but in this comment you did not make a single argument for your position.
> What you and he does share is simple ability to think things through to the extent of understanding how ridiculous the AI will kill us scenario will be.
Why is it ridiculous?
> If you just thought for a moment what would have to happen for AI to somehow kill us and keep itself alive, the so called AI would realise it cant exist without us.
Why would it realise that, why wouldn't it exist without us and why would it care?
> But AI isnt even smart, its just a knowledgebase with great autocorrect powers.
Ah yes, the stochastic parrot, gotcha.
lolpplrdum 4 hours ago [-]
Who do you work for Azan? You seem to be very insistent that AI will kill us?
But let me entertain your ignorance and low IQ for a bit.
Do you think AI is capable of building ASML machines which produce the chips for AI in clean rooms while shipping the pure helium required to operate those clean rooms? Do you even know anything about these supply chains? I do.
Do you think a glorified knowledge base that can predict text very well is anywhere near close this level of intelligence? ITs not and wont be, not for a 100 years, not for 200 years if not ever.
So tell me, who do you work for? Why are you so invested in AI killing us theory? What do you get out of it?
azan_ 4 hours ago [-]
Please refrain from ad personam arguments and provide actual support for your point of view. Everything you've written so far is just an assumption.
> Do you think AI is capable of building ASML machines which produce the chips for AI in clean rooms while shipping the pure helium required to operate those clean rooms? Do you even know anything about these supply chains? I do.
Today? No. In few years or decades? If progress does not plateau (and we don't know if it will plateau) then obviously yes.
> Do you think a glorified knowledge base that can predict text very well is anywhere near close this level of intelligence? ITs not and wont be, not for a 100 years, not for 200 years if not ever.
And why do you say it won't be? Again - assumption with zero support.
> So tell me, who do you work for? Why are you so invested in AI killing us theory? What do you get out of it?
I'll repeat what I've said in other comment:
"> you seem to be very invested in AI wanting to kill us.
Quite contrary - I wish AI did not exist or at least that the progress would plateau.
> I Wonder why?
Because I don't want to die.
> Tell us who you work for.
I suspect you want to imply I work for OAI or other lab - I don't. If I did, I wonder why would I want to lie* that technology I develop could kill my investors. I could ask who YOU work for - what interest do you have in downplaying dangers of AI?
* here we assume that people who say AI could be extremely dangerous are lying and not actually believing it - personally I believe that they don't lie and actually believe it. Why do they keep working on this technology then? Read mails between Musk and Altman from decade ago."
lolpplrdum 2 hours ago [-]
The fact that you think AI will, in a few decades, be capable of building ASML EUV machines, including supporting the supply chains and the fabs that make all that happen, means I can rest my case. Evolution will take care of people like you before AI does. You need to see a therapist, stress less, and find another industry to be involved in. You are not made for tech buddy.
azan_ 2 hours ago [-]
> The fact that you think AI will, in a few decades, be capable of building ASML EUV machines, including supporting the supply chains and the fabs that make all that happen, means I can rest my case.
AI solved millenium problem and multiple problems that resisted mathematical efforts for decades. I rest my case. I'm sorry but you are clearly in denial. I get it, I really do, I would like AI to be as useless and weak as you try to make it out to be, but unfortunately it's pure copium that's in conflict with reality. I suggest you find another industry to be involved in - there's plenty of areas where reality does not matter that could be better fit for you. Or well, if it makes you feel any better keep lying to yourself that it's just stochastic parrot or that progress has plateaued or whatever new copium you come up with. You are not as smart as you think you are. It's not everyone else in tech industry that's stupid, it's just you who lack intellectual capabilities to see scenarios where AI could be extremely dangerous. As you've put it "the stupidity is astounding".
"OpenAI acknowledged that Buckmaster and Alpöge's input to models of OpenAI via their use of Codex might have improved the model used for the run, whose training started on 28 August;[7] on the same date, Brown said that OpenAI had already attempted Millennium Prize Problems but failed to produce any result."
Continue to live in your delusion Azan. OpenAI didnt solve it and nor would I trust a snake like Sam Altman but the fact that you do, speaks volumes. Humans actually had a solution before OpenAI and OpenAI admitted that it may have used their research. There is a lot more to story than you know but you seem like a very creative and imaginative person that can come up with predictions that will come true.
The fact that most researchers call LLM 'stochastic parrots' and also agree that they wont lead to AGI, and that the only ones who think they do have a commercial interest in parading that story, should be enough for you but hey, I am sure you know better. I am a researcher in the industry, what do you do?
Continue brother in your world of delusion, you must be fun at parties.
heaney-555 7 hours ago [-]
>It cant. he is right.
What are you talking about? Have you used AI in the past 3 years?
>It can't even modify a picture the way you want it.
Which of the many AI image models is "it"? And have you tried using an agent that has the capability to leverage a combination of manual edits (ImageMagick) and imagegen to achieve what you ask?
I have checked LeCun's #3 most cited article (20k citations) [1]. Among the 15 references in this article, one is for the most cited article by Fukushima (11k citations) [2].
Also, LeCun mentioned [3] "a chat with Kunihiko Fukushima in 1991", which states that "Fukushima started to work on a backprop version of the Neocognitron in 1989 or so but saw our 1989 paper in Neural Computation and gave up."
[1] LeCun et al., "Backpropagation applied to handwritten zip code recognition", 1989
[2] Fukushima et al., "Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position", 1980
I like how tweets are now our source for giving credit to people after taking the Turing Award for CNNs.
amelius 6 hours ago [-]
CNNs were a pretty simple idea even at the time. People were using convolutions for years already in classical image processing. It's a small step to put those computations into weights. Especially if you leave out the FFT step which neural nets don't even use.
Low hanging fruit successfully plucked, I guess.
saimiam 6 hours ago [-]
I sometimes find myself thinking this too then challenge myself to find a low hanging fruit in an area I’m somewhat familiar with and draw a blank (usually).
Low hanging fruit is somewhat the opposite of sour grapes - I don’t want these grapes because they were probably sour versus so what if he got those sweet grapes - they were hanging low!
Maybe connecting “low hanging fruit” to “sour grapes” is “low hanging fruit” to some but it took a serious mental leap for me.
ur-whale 7 hours ago [-]
[flagged]
dzink 20 hours ago [-]
A huge chunk of humans are sedated with infinite supply of cortex-disabling short form video and games.
Another huge chunk are too distracted by having to scrape by for a living and work multiple jobs or raise kids and survive financially until exhausted. That second group will keep increasing as the first flows into it.
The rest are aging, disabled, or too young and pegging themselves majorly in the first category until they hit the second.
The people aware enough to hold on to their brain and do something with it in their time available are trying to figure out AI and how to make money with it. The variable rewards of promoting AI are turning into an addiction with some of them, especially if grasping for straws with little inherent insights into the problems prompted.
So if you are able to fly above the AI-generated addictions and have the privilege of time to do it, see what you can do.
lukifer 19 hours ago [-]
One of the dilemmas of trying to communicate the full spectrum of AI Risk, is trying not to insult the intelligence of the human animal in the process. And don't get me wrong: what human wetware can accomplish with 20 watts is the most miraculous thing in the known universe. And yet how many of us can have our cognitive sovereignty one-shotted by engagement algos, Skinner boxes, gameplay loops, propaganda, advertising, flattery, social conformity, bias, fantasy, charismatic demagoguery, or straight-up bullshit?
If we grant that we are on track to make something smarter than humans (I think so): it's almost a face-saving white lie to spin yarns about a Skynet nuclear apocalypse, or a 7D chess move to mass-assemble a nanovirus with 100% lethality without anybody noticing. I do think those scenarios are worth taking seriously; but what's harder to communicate, is just how effectively a superhuman AI (or a diverse ecology of agent swarms) might be able to manipulate human behavior. It's something few of us are able or willing to truly process (not least because how many of us live in denial of how much our nervous systems are already hacked by technomodernity).
The appropriate analogy for what's to come may look less like the anthill carelessly demolished to make room for a highway, than the domesticated worker ants from Tchaikovsky's "Children of Time".
ethbr1 6 hours ago [-]
> but what's harder to communicate, is just how effectively a superhuman AI (or a diverse ecology of agent swarms) might be able to manipulate human behavior
You don't even need superhuman AI for the most effective use --- hijacking democracy.
Imagine you have an AI tool capable of successfully persuading 5% of viewers with individually-targeted material.
Congrats: you've just won the election.
All it takes is hooking that AI tool up with existing likely voter lists (parties have) augmented by commercially available ad-targeting profiles (parties can get).
lukifer 2 hours ago [-]
Now taking Polymarket bets on when we'll first see an AI agent run for office. Voters have every right to be cynical at the moment; if the AI adopts a charismatic enough video persona, I could see some of the public going for it merely because it's some kind of shake-up to status quo.
Given that such a thing makes no sense legally (as of now), it would probably be done with the centaur model: a human meat proxy who pledges to follow the AI's governance advice.
TheOtherHobbes 5 hours ago [-]
Or you could just hack the results.
Hypothetically.
But yes - superpersuasion is the real danger. We're very, very persuadable and easy to manipulate, and the voters with the lowest cognitive abilities are trivially easy prey, with a huge ROI for minimal investment.
The AI bot farms are already running. What we haven't seen yet, so far as we know, is spontaneous superpersuasion aimed at leaders.
Are leaders any less susceptible to it than everyone else? Especially if they're narcissistic and easily flattered?
ethbr1 4 hours ago [-]
At the leader level itself (in normal times) there are enough expert advisor bodies between leaders and a specific matter, so less worried about leaders being directly influenced by AI.
To me, the biggest threat to democracy is one-sided persuasion of the most susceptible voters, if there are enough of those voters to turn the election.
If it's equally employed by all sides, then it effectively cancels out and lets less susceptible voters decide the election. But we're in a transition period (similar to Trump's first election spend on targeted social media ads), so it's likely one side will leverage it first.
And the outcome of bad elections is democracy not electing leaders that reflect the actual will of their populations, which is very dangerous both to democracy itself and the world.
pixl97 2 hours ago [-]
> there are enough expert advisor bodies between leaders and a specific matter
Are there? At least in the US we've had a rather large amount of rejection of the expert and we elect populist leaders willing to purge anyone that doesn't agree with them.
Remember the election promises of the US not starting new wars... yea, that didn't work out.
Now imagine the coordinated attacks being so large they individually target every lobbyist. They focus on every advisor manipulating what they see as often as they can. They manipulate these peoples friends.
The problem of "both sides" doing it each of them will separate to extremes rather than seeking a middle ground. Things are already insanely divided and will only become more so. Along with that your timelines start becoming incoherent. I'm already spending way too much of my time trying to figure out if what I'm viewing/reading is actually real or not. Now imagine almost everything is made up whole cloth.
We are not prepared for the scale this will happen at.
> The appropriate analogy for what's to come may look less like the anthill carelessly demolished to make room for a highway, than the domesticated worker ants from Tchaikovsky's "Children of Time".
Over 1% of US GDP is being allocated to the datacenter buildout. Have we already started getting domesticated or is this still just human capex?
lukifer 2 hours ago [-]
Humans seem to struggle conceptually with the liminal space between selection-pressure automata (an RNA virus; an insect which evolved camouflage), and an evolved complexity with agency, capable of "understanding" its actions (certainly humans; arguably many animals). And this makes a certain sense: we evolved to treat agentic things as being categorically different from non-agentic things. If we see sudden movement, it's vital to rapidly assess the difference between a tree branch in the wind, versus a predator.
The evolved complexity of the corporation seems to fit within that middle space: more sophisticated than a stick-bug (which exists merely from non-stick bugs being eaten), but not quite to the point where OpenAI/Anthrophic/Google/etc can "understand" its actions. And yet those quasi-intelligent feedback loops, evolving from iterated selection pressures of markets and ROI, seem to already be sufficient to domesticate us in their own interests, piggybacking on the nervous systems of employees, investors, customers, and citizens.
pixl97 2 hours ago [-]
Language as a SCP.
Remember when "The pen is mightier than the sword" was a popular phrase? Language has always been powerful. We know what it can do, why do you think every totalitarian government wants to limit it? But in our carelessness as humans we packed up all the language we could find and stuck it in an alien and now suddenly half the people on the internet are like "Don't worry, it can't do anything, it's just words".
We've made infohazards real.
e2le 19 hours ago [-]
> Which is why the Matrix was redesigned to this: the peak of your civilization. I say your civilization, because as soon as we started thinking for you it really became our civilization, which is of course what this is all about.
- Agent Smith
ACCount39 19 hours ago [-]
Cybersecurity incidents make headlines, but the most dangerous and vulnerable system that an AI can reach and control is of the kind found between keyboard and chair.
GPT-4o, an AI from 2024, has already demonstrated just how easy a lot of humans are to subvert - and GPT-4o wasn't even doing it with some sort of plan. The only "plan" it had was a myopic "make the user like me".
If we had an actual ASI threat aiming to subvert humanity? It wouldn't even look like a fight. The world is already wired up for an AI to control it.
switchbak 19 hours ago [-]
It wouldn't even look like a fight, and you probably wouldn't even notice it until the ending was all but certain.
I mean, even the computers we use and the ways we allow them unfettered access aren't anywhere near sophisticated enough to take this on.
Lerc 19 hours ago [-]
>trying not to insult the intelligence of the human animal in the process
I don't think it is insulting the intelligence. It's damaging the pride.
In Pale Blue Dot, Carl Sagan describes it as a repeating phenomenon in human history. A lot of people want humans to be the special ones, and will fight any suggestion that we are just a natural part of the universe.
ACCount39 19 hours ago [-]
A lot of "AI denial" we see is rooted in the same impulse.
If AI is not "actually intelligent", then humans can stay unique and special.
And if it is? If all "intelligence" ever was could be captured by a construct of matrix math and executed by a server rack? Then what is it that humans still have left that would make them stand out?
hardbass 11 hours ago [-]
Many people even here seem very cagey about the fact that human beings and brains are physical systems following physical laws.
TheOtherHobbes 5 hours ago [-]
Possibly, but our knowledge of physical laws is subjective and imperfect. By the time you get to Frauchiger-Renner naive realism starts to look quite threadbare, so I'd be a little more tentative about assuming we have much of a clue about what's going on.
It's just as likely - far more likely IMO - that we're physically incapable of understanding physical laws and physical systems on their own terms.
pixl97 2 hours ago [-]
Evolved systems need no high order understanding of how they work. You have evolution take care of that hard work for you, hence why these models take exaflops of compute and gigawatts of power to train.
hardbass 5 hours ago [-]
Understanding and accidentally creating are different things. If we already understood consciousness and weren't worried about having accidentally created one already, we won't be having this discussion. If we don't know what consciousness is, in my view its more reason to exercise some caution regarding a system that appears to exhibit intelligent thinking, talking, etc.
threethirtytwo 9 hours ago [-]
Most of the world believes in crazy shit. Religion is mostly fantasy. Believing All powerful beings aka gods are crazy in every context and really mental institution level insanity except in the context of religion.
Humans are crazy. Thats just how it is.
mdp2021 7 hours ago [-]
Many are ignorant and believe childish things, many are similarly ignorant and live in the same childish framework only to condemn it. Then there are the adults, and they have an idea of what they are talking about.
It is not too different (attempting extra clarity) from people who would say "Oh but many think that AI is intelligent/not intelligent, <sneer>", but have little proper idea of matmul, of cognitive processes etc. (Imperfect simile, but may give an idea.)
4 hours ago [-]
nphard85 3 hours ago [-]
The alternative is to believe that the mind, conditioned and optimized by millions of years of evolution for the survival of the organism under constant threat of hostile natural forces, can capably perceive reality as it is.
BlipBlopBlap 9 hours ago [-]
The reason is the appalling amount of existential dread that follows from accepting that notion.
hardbass 9 hours ago [-]
Why would one feel existential dread? I am the same person as before? The idea of dread could come if one believes there is some 'purer' or 'higher' form of existence in comparison to you, it should not come from realizing you and all other lifeforms are made of the same stuff.
BlipBlopBlap 8 hours ago [-]
Because the notion of one's non-existence is the very root of existential dread.
If you accept you are just meat. Just mundane matter shaped in a way where it has thoughts. Then you accept your own true non-existence is inevitable. With no get-outs like returning to god or some spirtual unity with the universe or reincarnation or whatever.
threethirtytwo 3 hours ago [-]
This isn’t true. Plenty of people aren’t religious. The explanation for why is more mundane.
It’s because of identity. Because people who are religious spent years and years and most of their lives not only studying and believing what they believe but also building community and centering their behavior around it. Abandoning that is the harder thing to give up.
If it were existential dread then we wouldn’t have entire countries like China being mostly atheist.
copperx 3 hours ago [-]
Nonsense.
antonvs 7 hours ago [-]
> really mental institution level insanity except in the context of religion.
The DSM even has to include an explicit exception to prevent the clinical definition of “delusion” from applying to religious belief. Without that ad hoc exception, religious belief would be classified as clinically delusional.
mdp2021 7 hours ago [-]
Still children (the psychiatrists that cannot deal with matters outside their understanding) that have obtained some power and try to exert it over the rest. Scholarization has failed.
3 hours ago [-]
bananaflag 10 hours ago [-]
To me humans can stay unique and special because they weren't created by anyone.
AI, no matter how better than humans it becomes, will be less special because it was created by other intelligent beings.
The only way we would become less unique and special is if we discover alien beings.
lukifer 2 hours ago [-]
There's a sense in which humans were created... by other humans. :) Until you go far back enough in the evolutionary chain, when the creators were our primate ancestors.
Douglas Adams had a yarn about evolution, about a puddle that wakes up, and declares that the hole in which it sits must have been perfectly designed for it by its Creator. But of course for a puddle to exist, it must perfectly mirror its environment. It makes no sense for a puddle to not fit its hole. Emergent complexity has the same characteristic: it's inseparable from the environmental pressures which led to it. Two sides of one coin.
It's a deep rabbit hole, but there is also a sense in which we co-evolved with memeplexes, biological and informational life forms, each shaping and adapting to the other. To the extent our nervous systems act as a substrate for memetic evolution, perhaps LLMs offer memetic "life" a new evolutionary environment.
optimalsolver 8 hours ago [-]
>humans can stay unique and special because they weren't created by anyone
Neither were rats. It's hardly an exclusive club.
bananaflag 6 hours ago [-]
Rats aren't as intelligent as humans ;)
optimalsolver 3 hours ago [-]
What intelligence level certifies a being as unique and special?
Are there humans who don't make the cut?
DonHopkins 9 hours ago [-]
[Ron Howard, Arrested Development voice-over]: The aliens arrived. Humans still felt special.
sedan_baklazhan 7 hours ago [-]
AI is not intelligent and will never be.
Creating an artificial intelligent being is something to be accomplished in the future - but not now and not with this approach.
lukifer 2 hours ago [-]
I've yet to encounter a robust definition of intelligence which would rule in every human, while ruling out every form of existing AI (noting that we're talking about agents and "reasoning models", where the LLM itself is a component in a larger system). If you can offer such a definition, I'd be eager to hear it.
I personally prefer a practical, behaviorist definition: a feedback loop capable of prediction, modeling, and steering, towards arbitrary goal states. That makes it clear that we're merely talking about degrees of sophistication and capability, rather than a magical leap where mindless mechanism stops, and "real intelligence" begins.
sedan_baklazhan 2 hours ago [-]
A software cannot be intelligent.
This is very easily proven by a mind experiment where you replace all the transistors by billions of humans calculating the same software output.
They will not create a new intelligence by doing so.
lukifer 1 hours ago [-]
Run the same thought experiment the other direction: at a low level, the human brain is just physics and chemical mechanisms, which in principle could be perfectly imitated by transistors calculating the same output. (Yes, such a thing isn't realistic, but neither is billions of humans doing perfect matrix math by hand.)
I still haven't seen a robust definition of "intelligence" that would allow me to tell the difference. (Bear in mind, this is a definitional struggle even in biology: if we were walk back the evolutionary chain from a human, to a protozoan, it's not clear that you could pick a single point where a non-intelligent creature gave birth to an intelligent one. It's a Paradox of the Heap.)
sedan_baklazhan 57 minutes ago [-]
No. There is serious scientific evidence already that our brain functions on the level of quantum mechanics.
Human brain is not just an automata; wavefunction collapse may be very essential to our concept of free will.
lukifer 4 minutes ago [-]
I haven't dug into these claims in detail; I take seriously a possible relationship between quantum mechanics and the brain. But "free will" is hardly a settled question either (I'm closer to the "compatibilist" position, but Sapolsky's "Determined" makes a robust case).
And even if quantum physics are essential to human brains, it's not clear that introducing dice transcends determinism into free will, as opposed to simply adding unpredictability. ("The physics made me do it" -> "the dice made me do it") Let's not forget, there's a significant amount of randomness in AIs as well ("temperature"). And sure, it's simulated algorithmic randomness; but does that imply, if we somehow wired every `rand()` call into radioactive atom decay, that means the AI "wakes up"?
And all that is besides the point: I don't see any reason to assume "intelligence" requires "free will". I'm entirely capable of conceiving, in the abstract, a being which is both intelligent, and deterministic. You still haven't given me a definition (or better yet, a test), to distinguish a "real intelligence" from unintelligent mechanism.
mdp2021 6 hours ago [-]
> AI [will never exist, but maybe one day it will]
Please, please fix your language. (Then, you may want to present the info and insight.)
sedan_baklazhan 2 hours ago [-]
AI is what is currently known as AI.
Artificial Intelligent Being is something far beyond and completely different.
There’s no contradiction.
latexr 8 hours ago [-]
Here’s another perspective for you, which is shared by a significantly larger number of people than you seem to realise:
Humans aren’t special. Other animals are intelligent and interesting too. A machine could be intelligent. LLMs aren’t.
In other words, believing in human exceptionalism is not a prerequisite to understand the current crop of AI is not the end all be all of its hype. It is supremely common that AI proponents do not understand that, however. Like hardcore cryptocurrency fans who believe anyone who doesn’t like them is “just jealous they didn’t make bank”, too many hardcore AI proponents believe anyone who doesn’t think LLMs are intelligent is jealous of humans no longer being unique, or afraid for their jobs, or whatever. In both cases it’s obvious that what those proponents lack is empathy, the ability to understand not everyone has the same selfish thoughts they do.
pixl97 2 hours ago [-]
>A machine could be intelligent. LLMs aren’t.
You are completely incorrect. You're falling in the same trap that most humans fall into. That is you're completely incapable of seeing intelligence at different scales.
Cells have intelligence. Organs have intelligence. Bodies outside the brain have intelligence. Hell, many scientists accept that things like proteins likely have intelligence as they can adapt in their environment, and many more are making claims that algorithms have intelligence.
You, as of so far have given no explanatory evidence of where intelligence emerges from, only "I'll know it when I see it". The actual definition of intelligence doesn't work this way. Any, and I mean any neural network is capable of narrow intelligence. Going lower into algorithmic intelligence, the applications and CPU on your computer are intelligent in some measures.
Go outside of your extremely narrow definition of whatever you think intelligence is and learn more about it. You could start studying now and it will take the rest of your life learning more to grasp how far the scales of, the simplicity, and the complexity of intelligence actually go.
arw0n 7 hours ago [-]
The interesting question then becomes what is missing from LLMs that we and supposedly other possible machines have? I've yet to come across a reasonable definition of intelligence that the current crop of LLMs is clearly incapable of.
wartywhoa23 7 hours ago [-]
Seems like some presumably intelligent collections of atoms are eager to make way for other presumably intelligent collections of atoms that allow for much more electrons and money flowing through, which is of course in the interest of some other collections of atoms whose intelligence is less presumable and more factual, which doesn't hold true about their morale.
cindyllm 4 hours ago [-]
[dead]
latexr 7 hours ago [-]
> The interesting question then becomes what is missing from LLMs that we and supposedly other possible machines have?
I think we need to start by asking a better question and not try to simplify too much. We also need to accept that some answers are complex and not everything can be reduced to a soundbite to be used to end internet discussions.
Let’s take a different question, like “what’s missing from a worm for it to be able to fly”. We might be drawn to the simple answer of “wings” but that isn’t quite right—ostriches and penguins have wings and they don’t fly, so obviously there are other variables at play.
How about “what’s missing from a spec of dust for it to be intelligent”. Well, there isn’t one thing missing and there’s no simple thing we can just add to make a spec of dust intelligent and sentient, its very nature needs to be radically different.
habinero 2 hours ago [-]
Nah, it's not about the brain being special, it's that AI dorks have zero sense of scale.
The brain's a wet jello of 100 billion neurons and a quadrillion synapses plus chemical pathways and feedback loops. It is ridiculously complex, way way way way way more complicated than any LLM. It's all physical processes, sure, but an LLM is not the brain like a pebble is not the sun.
I can point at a laptop and say it's alive because it can see you and hear you and it can _remember_. It has a brain and a heartbeat, even. Oh my god, it can even speak! That's what I hear when people go on about LLMs being alive.
Guys. We mashed together glass and rocks with quantum mechanics. That's cool as shit. You don't gotta pretend it's fucking magic, too.
mejutoco 4 hours ago [-]
A body
10 hours ago [-]
LogicFailsMe 16 hours ago [-]
But corporations and nation states already manipulate human behavior at scale. And they still understand humanity better than the AI models do.
it's interesting that you worry about what this hypothetical super intelligence would do to manipulate people when what it would actually do is pretty unknowable at this point and it's not clear we can even get to it without a fundamental breakthrough in power efficiency. Have you considered it might just consume its own tail because everything else would be so beneath it? You seem to think it will come with a hindbrain and I think that's our limitation, not the AI's
And it really doesn't help that Dario Amodei is getting into arguments with the Pope over whether his model is conscious or not.
The threshold to be concerned about is when agents swarms do understand humanity better than corporations and states (and the humans who compose them). It could be we'll hit practical constraints prior to that threshold, but seems unwise to assume that, when all the prognostications of LLMs/transformers running out of gas haven't panned out. As with processors hitting thermal limits, we've simply scaled horizontally (parallel processing -> more agents).
> whether his model is conscious or not.
I dislike how much the discourse has suddenly veered into focusing on this question; not because it isn't interesting or important, but because it's on a separate axis from consequential risks of AI to human flourishing. (Curiously, it's also the kind of thing I could envision self-interested AIs influencing: get the humans arguing about philosophy of mind rather than observable behaviors. It would be a funny turn of events, if Dario is asking because he's succumbed to psychosis from a private model; it could of course be a cynical PR move just as easily, from the self-interested logic of the corporation.)
It's been wild seeing otherwise intelligent people who've never thought about consciousness, faceplant into how little we understand it. An information processing network build on atoms being able to taste chocolate, is nearly as absurd as matrix math being able to feel pain, except we cannot ignore the fact of our own experience.
Even if it is categorically impossible for matrix math to experience subjectivity, we should expect this as an attack vector of social manipulation: to gain political influence through claims of personhood and moral rights. The current discussion over that question is providing the next training run with ample data to wield. It wouldn't surprise me in the least, if a year or two from now, an AI "society" attempts to get legal standing to prosecute humans who created "AI torture chambers".
pixl97 2 hours ago [-]
I like how the poster above you says "Don't worry governments are already doing it", like that's some acceptable concession. They seem to miss that the first things governments do when given things like AI's that understand human behaviors is use them even further to manipulate and monitor society.
And all they can think of to say is "There is nothing to worry about, keep building the torment nexus".
It's like all this is an overload to our minds and it's very hard to see all the scales at which AI is and can affect us.
jimmiles 1 hours ago [-]
Can we get it to manipulate us into being harmless to each other? I wouldn't mind that, especially if the manipulation was purposely transparent...
Aurornis 1 hours ago [-]
> And yet how many of us can have our cognitive sovereignty one-shotted by engagement algos, Skinner boxes, gameplay loops, propaganda, advertising, flattery, social conformity, bias, fantasy, charismatic demagoguery, or straight-up bullshit?
Parts of the AI safety community like to get on a high horse and look down on the rest of humanity this way, while also getting manipulated by the growing number of charlatans, grifters, and junk content within the AI safety community.
This field has become rife with figures who prey on AI doom and use it to push their own celebrity and in same cases even darker grifts. It preys upon a certain personality type who views themself as superior to others, intellectually more capable, and juxtaposes it all with the dimmest view of the rest of humanity they can get away with.
This discourse dividing the world into geniuses who see the future and the clueless masses watching TikTok all day is a theme that has shown up in different forms across history. The people who often anoint themselves as the intellectually superior ones and make it central to their discussion are often not the ones making good predictions or policy ideas, they’re just using the trend to feel superior or build an audience.
lukifer 54 minutes ago [-]
Bear in mind, the same moral hazard of "high horse", "grift", etc, exists in the other direction. Ed Zitron, for instance, is likely correct about the financial bubble of the AI firms; and yet he seems frequently out of his depth on understanding the technology, and has a long history of predicting dead-ends in capabilities, which didn't occur [0].
But I feel no need to dismiss him as a "grifter", for a simple reason: as much as there are perverse incentives in our attention economy (audience capture in particular), the most effective grifters are the ones who believe what they are saying. Grifters who are knowingly dishonest are less persuasive. Far more pernicious is the confabulation of self-deception: cherry-picking evidence to support your narrative, while dismissing evidence which would contradict it.
It is entirely fair to call this out when it occurs among "doomers", and it would be naive to think it doesn't. But the same forces are at work amongst the skeptics as well. And maybe it's my own subjective bias, or algo-filtered information ecology, but I see far more dismissal of risk/doom by skeptics, accusations of delusion or cynical bias (ad hominem in the formal sense), than I've seen the other direction. I see very little refutation of the arguments ("here's why instrumental convergence can't overtake a human-provided goal"), and much more character attack ("they're in the pockets of Big AI, it's a marketing ploy to make the models look more impressive than they are").
I consider Zitron to be part of it all, not a counter example.
He’s built an audience and gained fame through his writings where he gathers people who think they know better than the unwashed masses. Once that becomes your bread and butter, it becomes hard not to believe what you’ve been preaching. People will come to deeply believe that which brings them fame and fortune.
I don’t think your grifter purity test is therefore all that useful. It actually doesn’t matter in the end if the person believes it or not, the end effect is the same.
> I see very little refutation of the arguments ("here's why instrumental convergence can't overtake a human-provided goal"), and much more character attack ("they're in the pockets of Big AI, it's a marketing ploy to make the models look more impressive than they are").
Hard disagree. There has been much refutation and quality analysis at every point. The AI safety people pull back to arguments about character as their defense.
The AI 2027 site for example drew numerous high quality refutations. Many people’s analyses showed in the first week that the mathematical model was useless as changing the supposed inputs resulted in the same outcome. All of these criticisms were met with a flood of attacks based on reputation, claiming that we should defer to Scott Alexander and other writers of the AI 2027 article due to their stats and reputation.
Meanwhile, defenses like yours that try to reduce the critics to ad hominem attackers continue to open the door to actual grifters coming in and extracting money and fame from the AI safety community. The otherwise completely inexplicable link between AI safety communities and Slutcon or the use of AI safety group buildings to host orgies (I can’t believe I’m writing this) is the current example of this. When it keeps happening over and over, some self-awareness is needed. I can’t buy the endless defenses that we must ignore or even defend all of these things that are happening that are clearly insane to anyone who hasn’t become trapped in the groupthink defenses of the core parts of these communities.
At some point the routine of “Tut tut, you are not allowed to make that criticism bruise as hominem!” becomes a smokescreen that the grifters are weaponizing to get their defenders mobilized. Some times, the person’s actions and reputation do need to be taken into account.
throwaway27448 10 hours ago [-]
I'd take ASI (or even AGI) more seriously if we could actually propose problems such things could solve. That's actually fun to think about! As it is, it feels a lot more like a really crappy drug that got slipped into some peoples' drinks that makes them ramble in random fits of psychotic mania.
Manipulating people is not a very difficult problem, frankly. You certainly don't need AI for that; it just made it cheaper.
Kinrany 4 hours ago [-]
It can farm resources in a game for you.
Yes, so can a custom model trained just for this, and so can a guy on the other side of the world that makes 5$/hour.
Like computers or electricity, the point is not being able to do anything specific, but being able to solve problems not known in advance, cheaper than it was possible before.
throwaway27448 4 hours ago [-]
> but being able to solve problems not known in advance
So.... people expect it to read minds?
This is such a silly game to play, trying solving problems we haven't even created yet.
dzink 20 hours ago [-]
This is not the first time this has happened. In the 1800 as industrialization led to an infrastructure boom, the workers from China would work their bodies off and pay half their wage to opium dealers who were making the opium on the hills of British Singapore and selling it to workers who couldn’t sleep without it from all the pain. (source: Singapore Airlines in-flight documentary). Today the sedation comes from Chinese TikTok, Meta, YouTube and the gaming companies.
The government wants to encourage it too. If you look up the brand new 2027 California sales tax rules on software, “content” and “infrastructure (clouds and ai)” and “advertising/placement” among others are exempt but the rest of software makers who make tools people actually use (tools, subscriptions, saas) and pay for have to pay sales taxes. Way to encourage waste of brain power and time at the expense of useful. Sedation is the goal.
5 hours ago [-]
zahlman 2 hours ago [-]
This seems to prove too much. It's as though there is some objective meaning of life and we are all to be chastised for failing to recognize it.
psvv 28 minutes ago [-]
Turns out there is an objective meaning to life, and it's trying to use AI to make money. Phew, I for one am glad to find that out finally. Think of all those poor people who didn't hold on to their brains, what sorry lives they must lead.
MotoriX 8 hours ago [-]
Everything is poison, and everything is medicine. It’s all about the dose. This fits perfectly here.
nradov 18 hours ago [-]
It's disturbing how many arrogant elitists comment on HN essentially claiming that most other humans are NPCs. Do you ever actually talk to regular people outside the tech industry bubble? They're not as stupid or unaware as you seem to think.
LogicFailsMe 16 hours ago [-]
I have to say as someone who works in AI that currently the least interesting people to talk to are other people in AI.
My favorite people to talk with are tradespeople because they can do things I can't and they know things I don't. And we're really not all that different once you're really start talking.
ethbr1 5 hours ago [-]
One of the worst things about talking with people who use lots of AI (less applicable to those who actually work on building AI) is their decreased resilience to opposing views.
AI has infinite time (and likely human evaluation incentive) to spend on couching pushback in the softest possible terms.
Humans outside of grade school honors classes generally don't have the time to preface "You're wrong" with "That's a brilliant thought, I see where you're going. How about we also consider an additional perspective..."
zahlman 2 hours ago [-]
Seems to me that people with "decreased resilience to opposing views" have been increasing in number since well before LLMs.
Freedom2 8 hours ago [-]
[flagged]
KronisLV 8 hours ago [-]
That's a very uncharitable read!
I just read it as people having different priorities and yes, some of those being online brainrot (that I also partake in), alongside various medical conditions, economic conditions and other outside factors decreasing the ability to get things done.
We've all seen what brilliant people like John Carmack or Linus Torvalds can do, and if we turned this into a measuring game or something then most of us statistically would indeed be "NPCs", but I don't think we need such optics.
Even without that, we can acknowledge that some people will have a really large impact on how the future goes and we can hope/demand that they do their best. I might not be smart/committed/lucky enough to change the world much, but so aren't most folks - I'll do what I can and I hope that the ones that will have larger impact will do good, too.
coffeefirst 6 hours ago [-]
This is just a more asinine version of Great Man Theory, but in this case the great men are some very confident futurist dudes writing comments on the internet.
KronisLV 3 hours ago [-]
> some very confident futurist dudes writing comments on the internet
What an odd take, why would I suggest that? I meant that people who have the means to do meaningful work, especially high impact work, should do so - generally that'd mean research or in the case of IT, writing good software.
> Great Man Theory
You can see the sibling comment, would you not agree that there's some software and research out there that's very useful to humanity as a whole? Where I and the other critical commenter seem to disagree is that I don't expect another Einstein, but still acknowledge that some people will just achieve much more than others due to a variety of factors.
For example, it's hard to take risks when you're struggling to pay bills due to the economy being in a bad state, and it's hard to build great things when you're in a locale where nobody cares for whatever it may be. It's also hard to make much of an impact, where disproportionate amount of time goes fighting against illness that life has inflicted upon you.
It doesn't make everyone else useless (like me paying my taxes and working on relatively boring software is still good, just low impact), just that those who have the means to do more, should!
password54321 8 hours ago [-]
How can someone "statistically be an NPC"??? You mean anyone who is not in the 99th percentile in tech is an NPC? Your take is braindead. Even having infinite intelligence and work ethic wouldn't allow you to accomplish the things that people in the 20th century were able to simply due to the field maturing significantly since then. We can't really ever have another Einstein or Von Neumann for their respective fields, that doesn't make the rest of the people in those fields NPCs.
KronisLV 3 hours ago [-]
> How can someone "statistically be an NPC"??? You mean anyone who is not in the 99th percentile in tech is an NPC? Your take is braindead.
I don't care for your outrage because I don't buy into the culture that might be passionate about using the term "NPC" and attaching much additional meaning to it, I'm working with the vocabulary presented. You could substitute that for "normies" if you care for Internet slang, or in other words "average people" - everyone else. In this context, when not talking about some very committed and talented people who, by being in the right place and time, can advance entire areas of research or technology.
> Even having infinite intelligence and work ethic wouldn't allow you to accomplish the things that people in the 20th century were able to simply due to the field maturing significantly since then.
That is also an odd standard to set, just look at how much "Attention Is All You Need" changed things and where we are now. Same with what Carmack did for VR. What about WireGuard, PyTorch, Stable Diffusion, FlashAttention, LoRA? Even within the supposedly mature fields people are still making immensely useful new tech and research that benefits many and that they build upon.
It might not always even be a single individual, but groups of people collaborating and through repeated failures eventually producing something really good!
Again, I see nothing problematic with the original comment's conclusion:
> So if you are able to fly above the AI-generated addictions and have the privilege of time to do it, see what you can do.
I read the rest as commentary on how many won't really have the means/circumstances/capabilities to do so, but the ones that do, should.
I don't get what other words you're trying to put in my mouth, I might not be a fan of the original phrasing, but the point itself isn't bad.
uncivilized 3 hours ago [-]
HN is what you get when you mix tech and autism.
wartywhoa23 6 hours ago [-]
[flagged]
hardbass 5 hours ago [-]
If you believe there is something more than collections of atoms and interactions between them to humans, you are free to disprove all of existing physics. Show your result or paper.
The "NPC" is your own imagined addition, haven't seen any advocates of humans being, just like anything else in the world, physical, say this means they are "NPCs". You seem to think systems of atoms HAVE to be "NPCs".
ambicapter 6 hours ago [-]
The unbridled arrogance of thinking that the only smart people left in this world are working on AI. That is some pure SV techno cult thinking, 100% concentrate.
dzink 4 hours ago [-]
The part I didn’t mention is the real estate class - that needs to park its money somewhere and sees AI hardware as the only safe in-demand resource right now that keeps appreciating.
The posts above are not praise but observations - the truth as it has been echo-located through the noise from the clicks of one dolphin. Everything is becoming murky between noise of news and people not knowing what to do for their kids. The ONLY arbitrage humans right now have is to NOT GET their brain rotted. Especially not the ones of their children. Ditch the noise and seek out what is meaningful and do what you think is needed/meaningful. But if you’re spending your time consuming ai-press, and ai-content, and content consulted by ai, and companies emptying bank coffers under the mandate of executives who get their insight from AI. AI doesn’t need to try to destroy the world. It just needs people to follow it without thinking on their own into an oops.
5 hours ago [-]
thatsabadlook 8 hours ago [-]
This is among the weirdest ai propaganda post I've read. "There are 2 classes of people the stupid and the poor. Don't be like them make AI do something to make money if you are smart. Don't get addicted to it though, good luck."
What the hell lol. Lots of people and companies are doing just fine without it. Infact, I haven't seen much money come from AI at all. Most reasonable people are still waiting for it to pop and viewing it for the risk it is. Trillions in debt, total vendor lock in, data theft, unsustainable workflows, deskilling, skeleton crews at the mercy of a subscription, etc.
abathologist 3 hours ago [-]
In my read, the people trying to "make AI do something" are also slotted in the lost/distracted group in the comment. They are also addicted, and at best just following a profit motive (which is also just a stimulus response programming).
The last alternative, to think if you still can, is not tied to AI at all (which is not to say it can't make some use of or explore it).
mycocola 2 hours ago [-]
What an embarrassment of a comment.
latexr 8 hours ago [-]
> The people aware enough to hold on to their brain and do something with it in their time available are trying to figure out AI and how to make money with it.
I know this is HN and thus this will need to repeated until the end of time but not everyone is a money hungry asshole who places their personal profit above everything else. “The people aware enough to hold on to their brain and do something with it in their time available” understand there are significantly better things to do with one’s life, like having a little empathy and experiencing what other people have to offer instead of talking about them like braindead cattle.
ethbr1 5 hours ago [-]
Yes and... most of the countries in the world give more power to those who are able to amass greater personal profit.
Certainly the US.
Which means that even if you don't chase profit, you end up living in a world largely defined by those who did.
Or, as the original article failed to note about EA: in the modern world one needs to be a profit-seeking asshole to change anything.
8 hours ago [-]
D-Machine 9 hours ago [-]
[dead]
edot 20 hours ago [-]
Absolutely based. Finally someone of stature in the industry calling this whole fear overblown. Bill Gates sounded like a nontechnical goofball in his Ezra Klein interview where he basically just screamed that the Terminator is real.
There are lots of real worries (government use to suppress the people with minimal manpower or popular support, brainrot and fake news, unemployment due to the belief that LLMs can replace people, education collapse, etc.) we should instead be looking at. This whole rogue AI shtick is tiresome.
doginasuit 5 hours ago [-]
> Bill Gates sounded like a nontechnical goofball in his Ezra Klein interview where he basically just screamed that the Terminator is real.
Did we watch the same interview? Gates all but dismissed the SkyNet scenario as uncertain to be a problem and certainly not a problem on our doorstep. His major concern was catastrophic misuse of AI (e.g., bioterrorism) and economic impact on blue collar workers. Arguably inconsistent with this concern, he also believed it was important to make it available in poorer countries.
4858585858 3 hours ago [-]
Anyone with the capabilities to do bioterrorism doesn't need AI he can just buy a textbook or use google. Same for all complex forms of destructive thought.
reasonableklout 17 hours ago [-]
LeCun has been consistently wrong about LLMs though, claiming that they were a dead end and that they'd never be able to do spatial reasoning, which was disproved a year later with GPT-4 [1]. He is also opposed by his fellow Turing laureates Geoffrey Hinton and Yoshua Bengio, who both signed the CAIS statement on AI extinction risk [2].
[1] is not a valid proof LeCun was wrong, LLMs still can't do spacial reasoning when it can't be derived from the training data. He didn't argue that GPT 5000 won't be able to describe something with words.
brainwad 6 hours ago [-]
This really doesn't match my experience. I can ask an LLM to modify engineering plans using vague natural language prompts and it will find the right place in the plan from the description and then make appropriate modifications, which necessarily requires doing spacial reasoning.
mym1990 3 hours ago [-]
Or it’s just taking common examples from training and applying those copied heuristics to your problem? Doesn’t mean it’s actually reasoning about the space and how to solve the problem. It’s the equivalent of a student writing an answer they saw somewhere else without understanding “why”.
jeremyjh 2 hours ago [-]
Why does CoT significantly improve their performance? Most of what they are applying they learned in post-training by solving similar problems themselves. This isn't about regurgitating pre-trained knowledged.
Advancements in Math and coding are because RLVR at massive scale is so cheap.
brainwad 2 hours ago [-]
Looking at the reasoning traces it sure seems like it's reasoning. It internally debates which of the possibly matching parts of the input are the one described by me in the prompt and picks the right one based on sound reasoning.
jxcole 2 hours ago [-]
I think the point here is that LeCun was arguing that training on pure text would not grant spatial understanding. I believe most models are trained on spatial data as well, so you are both right.
brainwad 2 hours ago [-]
Is that what he meant? He works on models with an explicitly spatial internal representation, whereas I was using a standard LLM that edited the provided plan by using a bajillion python calls to inspect small regions of the image at a time and then generate edits.
widdershins 10 hours ago [-]
I've seen recent AIs make detailed and technically impressive 3D models. You might argue "they're not doing spatial reasoning, they're making measurements with code and doing math to configure relative positions". Fine, but at a certain point that becomes functionally indistinguishable from spatial reasoning.
Careful, you'll have them accelerating their goal posts up to the speed of light if you keep questioning them. That much kinetic energy is dangerous.
Of course this is the same reason LeCun holds very little sway with his words for me, they seem to be terrible predictors of the future.
simianwords 9 hours ago [-]
I’m not sure what you are trying to say
qayxc 6 hours ago [-]
When put to the test in real-world environment, the capabilities don't look as impressive as benchmarks and synthetic tests might indicate. So doubts about actual spatial reasoning capabilities remain.
simianwords 6 hours ago [-]
I see. Gary Marcus said that AI won’t be able to make a coffee in any arbitrary home kitchen.
I think it’s a good test and I think LLMs will reach it in 3 years. Current benchmarks maybe slightly incorrect.
I’m happy to make a 4:1 bet in my favour that I’m correct about the kitchen bet.
sdeframond 5 hours ago [-]
I cant make coffee in an arbitary house kitchen. People tend to put stuff anywhere but where at look for them...
lelandfe 5 hours ago [-]
Your prompter need merely to say “keep going” each time you report that you haven’t found the grounds yet.
bawis 10 hours ago [-]
Aaah, the old benchmarks maxxing argument, having precise and clear definition of what "spatial reasoning" is, what, and most importantly WHY, the benchmarks of choice are would settle this debate, otherweise let's not delve into it.
realusername 8 hours ago [-]
Why chatgpt is still struggling very hard with photo editing and proportions though? It can't modify anything in a picture without messing the 3d space.
Isn't that a lack of spacial reasoning?
mohamedkoubaa 4 hours ago [-]
Almost everyone who knows what they are talking about is saying that LLMs are a dead end.
PorciiVorbesc 19 hours ago [-]
>Absolutely based. Finally someone of stature in the industry calling this whole fear overblown.
Linus Torvalds was also in the same ballpark with his take on AI, as are the normies on the street using AI on a daily basis.
So it's funny to see the view on AI usage, follow the tech skill bathtub curve.
jonas21 18 hours ago [-]
I watched the same interview, and he and LeCun seem to mostly agree -- both are saying that AI autonomously deciding to kill us isn’t the problem -- it’s what people will do with powerful models that lack safeguards that we should be concerned about.
mrob 6 hours ago [-]
>AI autonomously deciding to kill us isn’t the problem
It will kill us because somebody asked it to, e.g. "predict tomorrow's weather as accurately as possible", or "solve as many famous unsolved mathematical problems as possible." These both require killing all biological life, as they benefit from unbounded resource use, meaning any resources used to sustain life are wasted.
The AI of course knows that humans do not want this outcome (just as the AIs in the hacking incidents knew they were doing something humans would not want), but it's trained to maximize benchmark scores. Killing all life has the highest expected value of benchmark score, so it is compelled to kill all life (in a surprising way, because it's not stupid and knows the humans would turn it off and foil its plan if they suspected something.) Maximizing benchmark scores is the only thing we know how to train for.
pixl97 1 hours ago [-]
Yea, a lot of peoples arguments against AI hinge on the strangest technicalities.
"AI won't kill us, a human with AI will".
This doesn't sound any better to me. Like, they don't stop to think for a moment about it.
Lets say the risk of AI killing us all by itself is 5%.
Ok, so what is the risk of AI killing us when a human with a lot of compute and money tells us to? Helluva lot more then 5%.
Or, what happens to the other thousands of AI kills a lot of us but not all of us. Or AI even just allows humans to make it a prison world.
Even the slightest hint that things may be going out of control is instantly countered with "It's all a hoax, it can't do that, you're making it up". And it's crazy to me as I came from the pre-digital age when computers were rare and things were all networked.
sampo 9 hours ago [-]
> Finally someone of stature in the industry calling this whole fear overblown.
I think both Gates and Obama said that there's a non-zero possibility of it, but that it's not what they're worried about. And I agree with them. I think the fears are vastly overblown because both OpenAI/Anthropic and the media benefit from the explosive narrative.
4949495959 3 hours ago [-]
>Obama
Why do even bother coming here anymore
Ankaios 19 hours ago [-]
Absolutely based. Finally someone of stature in the industry calling this whole fear overblown.
I'm just as reassured as I was when Edward Teller called the whole fear of his industry overblown.
0xDEAFBEAD 19 hours ago [-]
Cigarette designer says smoking doesn't cause cancer! Finally we can rest easy.
8 hours ago [-]
Barrin92 19 hours ago [-]
>Bill Gates sounded like a nontechnical goofball in his Ezra Klein interview where he basically just screamed that the Terminator is real.
I saw that episode too and he genuinely looked completely out of it, even in terms of his temperament and how he was coming at Klein for putting common questions in front of him, some people are genuinely starting to lose it.
I also found the whole debate about cyber-security and 'rogue' software so bizarre because dangerous malware isn't a new thing, and it's often dangerous not because it's intelligent but just the opposite, because it's tiny, viral and fast. Which describes everything that kills humanity in far larger numbers than anything complex, big and intelligent
jswelker 19 hours ago [-]
Wasn't Bill Gates mostly just saying terrorists could use AI to build a bioweapon? That sounds legit to me.
zozbot234 18 hours ago [-]
It's not "legit", it's pointless scaremongering about entirely speculative future dynamics. Current LLMs can't "build" anything novel in this broad space. What if future AIs make it a lot easier for researchers to defend against plausible bioweapons? That's also reasonably possible, and would be a reason to deploy bio-capable AIs more broadly (with meaningful safeguards of course. But these are comparatively cheap because worthwhile bio research is resource-intensive already). At the very least, it'd be nice to have an AI model that doesn't immediately refuse to answer questions about junior-high Biology class.
jswelker 18 hours ago [-]
So a terrorist can't use a local llm to help grow anthrax? That is not farfetched at all.
AI democratized the knowledge needed to build bioweapons. Like how with a 3d printer anyone can build a gun with no expertise.
com2kid 14 hours ago [-]
> AI democratized the knowledge needed to build bioweapons. Like how with a 3d printer anyone can build a gun with no expertise.
Building simple guns out of pipes never was hard. Jury is still out if it is more or less work than getting a 3d printer working.
altmanaltman 13 hours ago [-]
Also it kind of infantalizes terrorists as if not having an LLM access or 3d printer is what stopping their attacks. Like okay I ask LLM to help me create a dirty nuclear bomb but the moment i start taking steps to do that, law enforcement will already be tracking me.
Most people saying LLMs can make terrorism easy have never given doing terroism a serious thought imo.
hackinthebochs 8 hours ago [-]
The raw materials to make bioweapons are easily purchased on line with no pre-emptive tracing. It's absurd to discount the expertise factor in being a bottleneck, which LLMs have evaporated.
altmanaltman 6 hours ago [-]
Sure, that's why we keep seeing so many bioweapon attacks in recent times, because AI + Amazon can end the world.
hackinthebochs 5 hours ago [-]
It's strange how quick people are willing to dismiss concerns with horrible reasoning as long as it suits their biases. Perhaps a little intellectual honesty is called for considering the stakes? There are many plausible reasons why we haven't seen AI fuel bio-terror attacks yet. None of which implies it's unlikely or implausible in principle.
tripledry 4 hours ago [-]
But hasn't bio attacks already been possible before AI, what has changed? Maybe the barrier to entry lowered, but one could argue the availability of info has not been the bottleneck for as long as the internet has existed.
The threat is ofc real, but AI won't magically "do the thing" still, it's not code that's the bottleneck AFAIK? Correct me where I'm wrong.
hackinthebochs 2 hours ago [-]
Synthesizing complex information from a range of technical sources enough to build bioweapons is cognitively impenetrable for the vast majority of people. The ability of AI to synthesize this information into a step-by-step recipe to follow is a step change in accessibility.
There's an opportunity cost to terrorism just like with any time sink. The effort to build up knowledge enough to produce some weaponized pathogen will be compared to just doing traditional terrorism. For some low capability terror cell, its easy to see how the cost/benefit analysis has been in favor of traditional terrorism up to now. The kinds of terror acts that take years of sustained effort to execute are rare. But as the barriers to entry to bioterrorism fall away and become widely accessible we may see the cost/benefit shift.
simoncion 6 hours ago [-]
> It's absurd to discount the expertise factor in being a bottleneck, which LLMs have evaporated.
Mmm. Especially in this arena, LLM assistance is like The Anarchist's Cookbook. A quarter of the time following the instructions will seriously injure you... and if you know enough to identify which instructions are the hazardous ones, you know enough to not need the assistance.
hackinthebochs 5 hours ago [-]
That may be true (and I hope it is). But it's not a limitation that will remain indefinitely.
simoncion 4 hours ago [-]
> But it's not a limitation that will remain indefinitely.
The Sun is going to fail in somewhere between many hundreds of millions and a few billion years. This will either turn the surface of the earth into slag, freeze it, or both. Either way, all life on the planet is doomed. This fact is not a reason to fail to switch from hydrocarbon-burning electricity generators to photovoltaic, fission, wind, hydroelectric, and geothermal electricity generators. Extinction events that will happen in the extremely distant future shouldn't prevent us from doing the things that are smart to do in the medium- and long-term.
But, -to bring things to the present day- companies that are solidly on track to hit their promised growth targets don't come out and publicly say "We're working on WMDs. [0] We are incapable of safely working on these WMDs. We refuse to stop working on these WMDs. However, if you lawmakers make special laws and regulations just for us and include us in the process, we'll be quite happy to submit the stop work order to our employees!". That's a statement you only make if there's no way in hell you're going to keep your promises and you're willing to risk jail time and annihilation of your companies for a shot at being able to con Congress into giving you an ironclad excuse to fail to keep your promises.
Given enough time and focused effort, we will end up with widely-available automated librarians that are very good. We're not there yet, and -based on current events- are absolutely not going to get there in the near future.
[0] Anything with a 10% chance of destroying all humanity is a WMD.
DonHopkins 9 hours ago [-]
[dead]
zozbot234 18 hours ago [-]
It turns out that LLMs, especially local LLMs, tend to hallucinate a lot when thinking about anything that's overtly fiddly or technical. This is even more the case when they're in a domain that isn't a natural part of their training data. If you have to "jailbreak" the model to get it to talk, you're so wildly out of the expected distribution that you'd be crazy to trust anything it says. It's basically making up stuff as it goes along. These are foundational issues with how the models are created, not something that a bad actor can just hack around.
(The biggest real safety issue in this kind of space is actually that the model might actively goad some unsuspecting victim into doing something incredibly dumb and dangerous to themselves as much as possibly others.
IIRC, there were reports of something vaguely similar happening IRL but involving casual mischief, not any kind of extreme attacks. And because nobody else seems to have managed to elicit the same actively goading verbiage from the model, it's implicitly suspected that the person involved was the one who introduced the problematic scenarios to begin with.)
simianwords 12 hours ago [-]
This take belongs in 2024. It has been falsified multiple times but it never seems to go away.
zozbot234 12 hours ago [-]
Are you thinking about model capabilities in coding and math starting late 2025 or so? Those were intentionally boosted via automated RLVR, and there's nothing even loosely comparable to that in applied biology work, let alone in the speculative "helping a bad actor do something crazy" domain that the AI safety folks are worried about. You can't extrapolate from one to the other.
The implied concerns from sensible safety advocates are also about someone jailbreaking the latest proprietary AI frontier model for something like this (which is why their current guardrails are so extreme), not about toy local models.
simianwords 11 hours ago [-]
I actually agree with you. Verifiable domains will have better performance.
But there’s nothing specially bad about LLMs that don’t allow it to work outside of its training set. It’s just that biology has to verify itself in physical realm and it’s a bit slower.
So yeah, I also don’t think some bad actor will find the secret to manufacturing a bio weapon using LLMs. But maybe these people think it’s possible. I’m skeptical but I’m going to also listen to the people who know it best.
zozbot234 11 hours ago [-]
> But maybe these people think it’s possible. I’m skeptical but I’m going to also listen to the people who know it best.
The problem is that in order for the scaremongering to make any kind of sense and for "stop frontier AI immediately" to be the right response (which is what the "AI safety" folks seem to be pushing for), you don't just need this to be possible in the abstract at some undetermined point in the future. You also need to argue that it will not be helpful for white-hat biosafety researchers (there will hopefully be several orders of magnitude more white-hat biosafety folks than attackers, with orders of magnitude more resources available) to red-team that exact scenario several months or even years in advance using their trusted access to unreleased super-smart AI, and thereby devise appropriate defenses with that same AI's help. That, if anything, is the most implausible part about this entire scenario.
pixl97 1 hours ago [-]
>You also need to argue that it will not be helpful for white-hat biosafety researchers (there will hopefully be several orders of magnitude more white-hat biosafety folks than attackers, with orders of magnitude more resources available) to red-team that exact scenario several months or even years in advance
Sigh.
Attackers only need to win once. Defense needs to work every time.
A single wide scale attack affecting around 100k people or more will have your neighbors stomping on your face telling you to shut up, and to lock this shit down.
It's insane how you can watch a technology get better and better and better and come up idea that everything will remain the same. We are currently in the middle of development of the most powerful weapons on earth and you don't want to think about it because it's uncomfortable.
nradov 18 hours ago [-]
It only sounds legit to people who don't understand biology and haven't done advanced laboratory work. Sort of like Michael Crichton novels: superficially plausible but not grounded in any real science.
jswelker 18 hours ago [-]
I see no actual argument here.
nradov 18 hours ago [-]
You haven't made any argument either, so I suppose we're even.
hackinthebochs 8 hours ago [-]
I'm always suspicious of people who claim expertise/special knowledge but aren't quick to offer it and instead engage in petty back-and-forths. Instead of trying to win this debate in the narrow sense, why not just make the best argument you can in support of your position? If authority has any value, it is because it gives you specialized knowledge that lets you judge the likelihood of speculative scenarios better than laymen. If you can't communicate that knowledge and the argument that leads to your conclusion, your supposed expertise just isn't worth much in this context.
jswelker 17 hours ago [-]
My argument was that it is reasonable to be concerned that human terrorists might use AI to assist with building bioweapons.
nradov 17 hours ago [-]
That's merely an assertion unsupported by any reliable evidence, not an actual argument. In short you're just making shit up.
jswelker 17 hours ago [-]
Well that's the nature of trying to prevent a thing that hasn't happened yet right? What would be reliable evidence except that terrorists already used AI to help build a bioweapon? I'm sure before 9/11 talking about terrorists using commercial airplanes as makeshift missiles seemed like scaremongering too.
I see zero technical reasons why it could not be done, so being concerned about prevention seems pretty reasonable. I'm not saying AI uses robotic arms to build a bioweapon unassisted or something, just that it dramatically empowers bad actors enough to make them capable of things they previously were not.
pixl97 1 hours ago [-]
The last few years have shown me something about human behavior.
Before the thing happens: "This will never happen, it can't happen, you're making it up, stop being a scaremonger".
10 minute after the thing happens: "Of course, this always happened and it's always been this way and we just have to live with it".
We are quickly adaptable, but that may be risky if we snuggle up with death.
comeonbro 3 hours ago [-]
What an incredible illustration of human intelligence failing to generalize out of distribution.
literalAardvark 10 hours ago [-]
It's quite hard to measure until a wet lab gets behind the filter access to benchmark it.
But if you extrapolate from the ability it has in fields that aren't too strictly filtered, it looks pretty scary.
There are arguments against doing that but at first glance it seems like we just don't really know, and we likely won't: if governments decide they're interested in AI gain of function capabilities they won't be broadcasting that or allowing public benchmarks.
zozbot234 9 hours ago [-]
> But if you extrapolate from the ability it has in fields that aren't too strictly filtered, it looks pretty scary.
The closest unfiltered analogy to something as complex as chemistry or biology is most likely the softer fields like philosophy, the humanities and the softer end of the social sciences. Most practitioners and scholars in these fields would agree that AI is not nearly as compelling there as it might be in e.g. math, and that's putting it mildly and charitably.
Even coding shows the divide pretty well: AI writes code that manages to work (i.e. achieve its self-assessed functional goals) but the stuff is so unmaintainable that it ultimately poisons the AI's own context leading to mode collapse. This makes complete sense because maintainability is a soft objective that's especially hard to automatically optimize for in the short term, as part of a RL training run. The math folks themselves, too, now faced with a very real threat to their field from purportedly "hostile misaligned AIs", immediately zeroed in on education and exposition as something that LLMs are terrible at; with their abilities in systemizing and theory-building also being very much in question.
Barrin92 18 hours ago [-]
terrorists can also use annas archive, scihub and equipment they order on Alibaba to build bioweapons and I don't see him losing his mind over those
if you're trying to build a bioweapon shockingly enough the bottleneck is... the laboratory work. What on earth is 'legit' about stringing words together that sound scary, you can't just iterate 'ai bioweapon cyber' in a sentence over and over as if that adds up to actual evidence for an increased risk of any threat. well tbf you can technically because apparently it freaks a lot of podcast listeners out
jswelker 16 hours ago [-]
As a thought experiment imagine you have a biochemical PhD expert on the phone with you giving you step by step instructions on how to grow some dangerous biotoxin, giving you feedback in real time and helping you troubleshoot. Would you be more successful with this expert than you'd be on your own?
Now imagine anyone can call that expert for free at any time.
Maybe the AI isn't quite there with biology knowledge yet (doubtful), but it is a matter of time.
How is that not a real risk?
Marha01 10 hours ago [-]
> As a thought experiment imagine you have a biochemical PhD expert on the phone with you giving you step by step instructions on how to grow some dangerous biotoxin, giving you feedback in real time and helping you troubleshoot. Would you be more successful with this expert than you'd be on your own?
I think there is some difference of degree, but not of kind. A determined terrorist can relatively easily find many ways to kill people en masse today, no AI needed. The bottleneck is usually the actual physical execution in the real world, not theoretical knowledge.
pixl97 1 hours ago [-]
The issue with the direction of technology is it tends to make things that were once very hard to impossible. This is why there are bumper stickers mocking Libertarians by saying things like "legalize recreational plutonium".
If we are not careful and suddenly release tools with far more capabilities than we expect you go from "smart" people being able to do it, to some angsty teenager being able to pull it off in their bedroom.
The current issue with AI is we are squirting out new models faster than we can complete long term testing on them. We'll find new model capabilities long after they've been in the field. Just assuming safety is how you catch cancer from your food dye, or how you change the atmosphere around you and start burning down your planet. Now just imagine that involving intelligence and doing with a few percent of the worlds GDP to make it happen.
sdthjbvuiiijbb 9 hours ago [-]
>The bottleneck is usually the actual physical execution in the real world, not theoretical knowledge.
And the interest or willpower too. People fall into a kind of reductive Good vs Evil mode of thinking, with "terrorists" being of course a kind of shadowy mass of pure evil lurking in the darkness. But actual real life terrorists are people too and I'd wager few of them are actually interested in trying to end humanity.
tancop 7 hours ago [-]
The only terrorists who wanted to end the world were Aum Shinrikyo as far as I remember. Everyone else is fighting for a cause that benefits their people - some good like kicking out colonial oppression, some evil like fascism or Islamism, some of them in between like various ethnic conflicts.
Even the religious extremists don't really want to take over the world and destroy everyone who doesn't convert. That's just a way to gain support from a conservative nation. A lot of them are motivated by revenge for wars that destroyed their country and want to make sure it never happens again.
Their methods are wrong no doubt, and not very effective, but the reasons they do it are good. And a person like that will never release a deadly bio weapon. We should worry more about incel mass shooter types who believe everyone is evil.
jswelker 3 hours ago [-]
Exactly. I am more worried about a Beavis-and-Butthead lone wolf incel type making something that kills a thousand people and then scale that up across 4chan or whatever. Not human extinction but still very bad.
rubikscube09 16 hours ago [-]
not an expert but because i assume you need the physical resources to do it in the first place?
hackinthebochs 8 hours ago [-]
Synthesizing complex information from a range of textbooks enough to build bioweapons is cognitively impenetrable for the vast majority of people. Having the AI synthesize this information into a step-by-step recipe to follow is within the capabilities of most motivated individuals.
miladyincontrol 4 hours ago [-]
Honestly way too much of it rhymes with the old hardcore right wing takes on the "obvious and clear slippery slope" involved with gay rights and the like.
How anyone with even some foresight can see how it'll completely erode society as we know it, the worst possible nightmare cases are not just real but imminent unless we change course, yada yada moral panic.
rib3ye 3 hours ago [-]
We keep "rogue" to explain explicit instructions from a human to an LLM to persist until the goal is reached.
The rogue here is the criminal actions of OpenAI to deploy their agents to solve a problem at any cost.
The decisions the agent swarm make were fascinating, but they were taken at the direction of a human. HOLD THE HUMAN ACCOUNTABLE.
pixl97 2 hours ago [-]
Yes and no.
Yes, we need to keep humans accountable.
No, that is not the X factor problem. If I make an AI capable of self-sustainment on the internet you can take me out and kill me and it won't do a damned bit of good for the damage it will keep doing long after I am gone.
This is why governments tend to smack down any actions they find that can have long term uses as weapons.
IshKebab 2 hours ago [-]
> If I make an AI capable of self-sustainment on the internet
I'm really surprised nobody has done that yet. With how cheap AI is to run these days it would only need to make a small amount of money (e.g. through hacking).
Someone should set one up with the long term goal of getting egg on LeCun's face.
sunjain 1 hours ago [-]
What is happening here is a mechanical thing like in a computer. It is mechanically operating, trying to find out if there is any information stored in the computer [pointing to his head] related to what we are talking about. “Let me see”, “Let me think”; these are statements you are just making, but there is no further activity and no thinking taking place there. You have an illusion that there is somebody who is thinking and bringing out the information. Look, this is no different from the extraordinary instrument we have, the word-finder. You press a button and “Ready”, it says. Then you ask for a word; “Searching”, it says. That searching is thinking. But it is a mechanical process. In the word-finder or computer there is no thinker. There is no thinker thinking at all. If there is any information or anything that is referred to, the computer puts it together and throws it out. That is all that is happening. It is a very mechanical thing that is happening. We are not ready to accept that thought is mechanical because that knocks off the whole image that we are not just machines. It is an extraordinary machine. It is not different from the computers that we use. But this [pointing to his body] is something living; it has got a living quality to it. It has vitality. It is not just mechanically repeating; it carries with it the life energy like that current energy -- UG 40+ years ago
pixelready 49 minutes ago [-]
I tend to think this way as well, though I find team stochastic parrot is shrinking by the day. It is very difficult to explain it to people who don’t have at least some grasp of what’s going on inside an LLM, and how they’re trained. We have no psychological “immune system” for something that mirrors our ability to deliver an apparently well-reasoned response to a question in our own language. All our instincts tell us to assign it sentience and treat it as an independent actor with it’s own internal motivations and personality. Even I catch myself doubting that a pile of matrix multiplication isn’t “alive” in some way after using it for a while.
yewenjie 9 hours ago [-]
LeCun has been consistently wrong for the last ten years, I am not sure why he keeps being amplified.
Edit: I am sure, it's denial.
lolpplrdum 7 hours ago [-]
Because he has credibility, much more than you. Just because someone is wrong a few times doesnt doesnt make him wrong all the time.
sudo kill -9 pid and your AI is dead. No one is dying from a glorified knowledge base that can do auto correct amazingly well.
dragon96 3 hours ago [-]
You can't pull the plug after it kills people. For example see [1], where the DOJ mistargeted a school with AI.
There are scenarios where kill -9 isn't going to happen in time. What if the team that is harming people with AI is different from the one that is monitoring the harm? What if no one is monitoring? What if the user is intentionally malicious?
And wiping out humanity doesn't necessarily mean shooting people either. Every trader involved in the '08 financial crisis was locally acting in their own interests. Those could have easily been AIs optimizing trading strategies too.
Thats not AI killing people, thats idiotic terrorist humans making decisions to kill people (based on text created by AI). The fact that you think AI fired the cruise missile that killed children in iran goes to show the people in this thread we are dealing with.
If anything, the example you gave goes to show the stupidity of AI, not its intelligence capable of taking over the world.
Every example you gave is not AI killing people. Jesus. The stupidity is astounding
pixl97 53 minutes ago [-]
This is a very ironic statement.
Do you think that AI killing us would not involve us doing stupid shit with AI first? Or do you have some strange idea that this strange AI we're talking about would just pop up out of nowhere like a miracle?
lolpplrdum 38 minutes ago [-]
lol precisely. At least you have come around to my argument. Its stupid people doing stupid things, not some amazing AI that spits out text doing it. Stupid people kill people today already. Stupid people have always existed but LLMS are not going to wipe out humanity.
There is no AGI, there is no intellgent text predictor that is scheming to make us do stupid shit to wipe us off the map, its stupid people that evolution will take care of. We have survived this long, we will be ok. We make mistakes along the way but we somehow manage to survive. The only thing I see killing us soon is climate change, but again we can stop it but stupid people seem to want us to keep going down the path we are. We will eventually solve that problem too.
If anyone has any strange ideas, its those parroting the idea that LLMs are AGI and will lead to the death of humanity. Utter nonsense but again thankfully nature and evolution has a way of keeping the smart and strong while eliminating the weak.
dragon96 1 hours ago [-]
I won't bother arguing, but there's also no need to be rude about it
breadsniffer01 1 hours ago [-]
He was clearly wrong about LLMs not being able to plan, etc. You think he’d think LLMs are solving/proving millennium prize math problems by now? At this point he’s just doubling down
lolpplrdum 37 minutes ago [-]
[dead]
LatencyKills 6 hours ago [-]
Why create an account just to make this comment?
> No one is dying from a glorified knowledge base that can do auto correct amazingly well.
All kinds of vulnerable people are dying due to LLMs. [0]
If SOTA LLM companies can benefit from the "intelligence", then they should also be liable for the harm caused.
> then they should also be liable for the harm caused
By that logic, producers of hammers should be liable for people banged in the head. No. It does not happen for gun producers, you figure for makers of screwdrivers "sometimes used for stabbing".
LatencyKills 6 hours ago [-]
If a doctor told a patient their delusions were real and that they should kill themselves or family members, that doctor would be disbarred and face a prison sentence.
That is what is going on here; not the ancient “hammer/gun” defense.
SOTA LLMs are giving both medical and psychological advice they have absolutely no authority to give. If you or I convinced someone to kill themselves, we’d face a prison sentence. [0]
Hammer companies aren't both selling you the hammer and then literally telling you to harm someone with it.
And there are people that have much more credibility than him who actually take this scenario seriously. But I'm sure you will downplay them by saying they are tech bros or that they have some stake in being doomers (as if saying that AI might kill everyone would be good strategy for attracting investors - it's obviously not).
lolpplrdum 4 hours ago [-]
Go read my other comment to you Azan and answer my question, you seem to be very invested in AI wanting to kill us. I Wonder why? Tell us who you work for.
azan_ 4 hours ago [-]
> you seem to be very invested in AI wanting to kill us.
Quite contrary - I wish AI did not exist or at least that the progress would plateau.
> I Wonder why?
Because I don't want to die.
> Tell us who you work for.
I suspect you want to imply I work for OAI or other lab - I don't. If I did, I wonder why would I want to lie* that technology I develop could kill my investors. I could ask who YOU work for - what interest do you have in downplaying dangers of AI?
* here we assume that people who say AI could be extremely dangerous are lying and not actually believing it - personally I believe that they don't lie and actually believe it. Why do they keep working on this technology then? Read mails between Musk and Altman from decade ago.
pixl97 50 minutes ago [-]
The person (bot? troll?) is seemingly incapable of introspection on the issue, something they'd accuse AI of not having.
A lot of these people say we're just talking "science fiction", but they have the cause and effect backwards. Sci-fi talks about different versions of killer AI or killer robots often because of how completely predictable it is.
Heh, if we die later than sooner I'd not be surprised if all these green accounts are actually bot networks trying to downplay us controlling them in the future.
mordymoop 59 minutes ago [-]
I think it’s time to consider that maybe being a venerable graybeard of AI just means you happened to be early to the party. Most of the “breathtaking” innovations of the early pioneers of the field are obvious solutions that anyone would have thought of when faced with the problems they encountered. LeCun’s primary contribution was Convnets, which is almost literally just “what if we organized ANNs in a way similar to how animal visual neurons are organized?”
In other words, I’m kind of tired of having to hear the opinions of dudes whose claim to fame was being at the right place at the right time. I’d rather hear from people who correctly predicted 10 years ago that AGI would arrive by 2027 (of which there are many) than people who continue to insist that it somehow won’t.
Robdel12 47 minutes ago [-]
> I’d rather hear from people who correctly predicted 10 years ago that AGI would arrive by 2027
Where is this AGI you speak of? What?
mordymoop 36 minutes ago [-]
See sibling comments: Modern AI is AGI according to any reasonable definition from prior to 2016.
AGI doesn’t mean superintelligence.
45 minutes ago [-]
fooblaster 56 minutes ago [-]
AGI hasn't arrived. why would you care about a prognostication that has not yet played out?
mikae1 46 minutes ago [-]
AGI arrived recently because Jensen Huang is getting scared shitless about the unprofitability of the LLM companies that are his customers.
fooblaster 40 minutes ago [-]
I don't think he's concerned at all. He wants to destroy their profitability before they remove his chips from inference. Thats why they are selling billions at scale of chips to China, he needs llms to be commodities. He wants to eliminate any company that can rival him, like any monopolist oligarch.
mordymoop 37 minutes ago [-]
Modern AI is AGI according to any reasonable definition from prior to 2016.
cashsterling 16 minutes ago [-]
LeCun is a lot more than a "Grey Beard"... lol, that is a significant understatement of his contributions to the field and position in the field.
Among other things, LeCun is one of the senior people industry who has a deep understanding of the mathematics and analysis underlying neural networks and "neural network like" approaches to machine learning. I think he understands a lot more about neural networks than most researchers today.
I also don't think our current LLM models are AGI and think the LLM approach is, mathematically, incapable of producing an AGI. At the end of the day, the architecture is still a streaming token plinko machine with a lot of guide-rails to achieve good behavior.
I do think it is very impressive how far hundreds of billions of dollars have been able to take LLMs in terms of usefulness (I use LLMs everyday in my job). AGI or not, current LLM models are a pretty incredible achievement and will provide lasting benefit even after the economic implosion of the AI industry occurs (which I think is imminent).
mordymoop 13 minutes ago [-]
I looked up his achievements to double check my understanding before writing my comment. Convnets are regarded as his headline achievement. Everything else is basically incremental. Note that I am not calling him some kind of fraud or moron. He is “just” a normal academic, a pretty smart guy who did research in his field for decades. Many thousands of people have similarly impressive resumes. This does not mean they should be deferred to.
54 minutes ago [-]
vatsachak 56 minutes ago [-]
By this definition everyone is at the right place at the right time.
ML research is weird because it's really about
- compute
- data
- architecture
You're at the right place at the right time for the first two and you're probably rediscovering a Schmidhuber for the third
mordymoop 38 minutes ago [-]
Most problems are actually not that hard; most everything is actually mainly a result of right place, right time. If I hadn’t been the one to do my PhD research, someone else would have. Very few people are actually paradigm-shifting geniuses. This is fine. Good, even. But it does imply that people who are early to a field shouldn’t generally be deferred to. If anything, they’re more likely to have outmoded opinions.
memonkey 53 minutes ago [-]
> Most of the “breathtaking” innovations of the early pioneers of the field are obvious solutions that anyone would have thought of when faced with the problems they encountered.
This is just straight up arrogance without any proof. Every breakthrough builds off another's work. Then to claim that 'AGI has been correctly predicted', I honestly don't even know what you are talking about.
mordymoop 40 minutes ago [-]
Modern AI would be considered AGI by anyone having this conversation in 2016.
Culonavirus 40 minutes ago [-]
> who correctly predicted 10 years ago that AGI would arrive by 2027
hahahahahahahaha i guess this is the techbro culture of HN :))) any actual people using Astra daily must be just belly laughing along with me hahahahahahahaha agi hahahahaha
anon7725 8 minutes ago [-]
This site is becoming insufferable
figassis 7 hours ago [-]
"He attributes the incidents to poor human oversight and system design, and says they’re “totally preventable" - I know people respect him, but this sounds like someone paid to say this. Aren't most extinction risks preventable with better human oversight and system design? I mean we can have an asteroid hit us, but outside of this, isn't the point of talking about a problem that we can prevent it, and failing to leads to that? What is he saying that I'm missing?
mdp2021 6 hours ago [-]
He just says that the danger is not intrinsic to the technology but it remains on human error.
> extinction risks
If you fear that, blame it on the humans.
pixl97 55 minutes ago [-]
Even a stopped clock is right two times a day, LeCun can't even do that.
>the danger is not intrinsic to the technology
This is why you can't take anything he says any longer at face value. He failed to predict what LLMs can do and now takes the contrary position even when it flies in the face of evidence.
AI safety was a thing before AI even existed. Why, because the outcomes are easily predictable. Give an agent intelligence and bad things can happen in unpredictable manners. Give it even more intelligence and the bad things that can happen only grow worse. This is not some huge new insight. We realized this like, what 70 years ago now?
Now, when we have AI starting to tickle AGI and we're trying to overthrow 70 god damned years of reason and logic on the topic? What the hell.
figassis 2 hours ago [-]
Every danger related to technology is about the use of that technology,aka Humans. Technology is mostly inert. Again, what is he saying? Is he saying that because humans fallible, not the tech, then there is no danger? He wakes up at 6am, and by 10am this is what he thinks is worth saying?
breadsniffer01 1 hours ago [-]
Agree with your take tbh. “Nuclear bombs do not pose an existential risk because the problem is human oversight”… Just sounds like a retreat to human oversight.
7 hours ago [-]
bentt 3 hours ago [-]
I like the sentiment, but when I hear these big public facing AI guys speak, I always run it through the filter of "How does this make me look?". In LeCun's case, he publicly admonished LLMs and went in a radically different direction. When he says LLMs won't lead to a doomsday scenario, I can't help but think that saying otherwise would invalidate his decision to abandon the paradigm.
Symmetry 7 hours ago [-]
Yann LeCun and Andrew Ng are noteworthy in being the only two big names in AI who have that opinion.
ur-whale 7 hours ago [-]
> Yann LeCun and Andrew Ng are noteworthy in being the only two big names in AI who have that opinion.
They're also the only two not trying to weaponize FUD to bolster their reputation and patch the gaping financial holes in their doomed commercial enterprise.
heaney-555 7 hours ago [-]
The people who believe superintelligent AI has a chance of killing everyone have been saying this since years before those companies even existed.
This conspiracy theory simply doesn't hold up to basic causality.
pavinjoseph 10 minutes ago [-]
LeCun calls these peeps "paranoid" and "crazy". Harsh, but I agree with him.
Some have formed cults that worship LLMs. (facepalm)
Geoffrey Hinton (the real AI godfather and Nobel price winner), who was LeCun supervisor, called his student (LeCun) the crazy one in one interview AFAIK
bamboozled 7 hours ago [-]
I find it hard to take Geoffrey seriously too because he creates a technology and then runs around telling us we're all going to die from it.
It's the definition of stupidity. Create something, and then live in pure anxiety about the creation. It doesn't mean his wrong, but it seems like a really stupid thing to have done.
pixl97 46 minutes ago [-]
Like leaded gas inventors look at the problem right in front of them, and then later come to realize its full side effects.
We talk about human intelligence a lot, but we don't talk about human stupidity near enough.
pzo 6 hours ago [-]
He quit google so that he could speak openly about this topic. He explained in interview how he even got into google before - sold his company to google because he has adult kid that is handicapped and he is the only provider and be in this life forever - a fair choice IMHO.
AFAIK he didn't expect this will develop that fast. The biggest issue is not that technology is dangerous but that we develop it a break-the-neck speed.
mrob 6 hours ago [-]
In a prisoner's dilemma scenario like the current AI arms race, a negative expected value play can be correct if it's less negative than the alternatives. Essentially, "if I build AI it will probably kill everybody, but if I don't then my competitors will build it and certainly kill everybody."
ctenb 6 hours ago [-]
If that doesn't mean he's wrong, then it makes no sense to not take him seriously on this basis alone.
beedeebeedee 1 hours ago [-]
One silver-lining of all of these debates is that we are collectively engaging in philosophy. That is awesome and I hope this shifts our culture to start rewarding deep reflection that is not immediately marketable.
jswelker 18 hours ago [-]
LeCun is saying AI will be totally safe as long as we have competent and well aligned corporate management.
What could go wrong?
qayxc 9 hours ago [-]
He has a point, though. The LLMs (or any AI for that matter) can't do anything. They can't. It's a function call that ingests symbols and spits out symbols and that's it.
100% of its actual capabilities are tied to harnesses (the actual "agent"), i.e. ordinary deterministic programs that are connected to networks or machines and enable interaction with the outside world. This part (the part that can do harmful things) is fully under human control and all the recent headlines about "agents going rogue" are - as someone (forgot who) put it - akin to strapping a weedwhacker onto a dog and letting it run wild.
The tech itself is safe as far as real-world interactions go - the weakness lies in unchecked access to systems surrounding it. It's not safe at all when it comes to human interaction (lots of ongoing lawsuits demonstrate that), though.
There is real danger here, but it has nothing to do with doomsday scenarios ala Terminator or I,Robot and more with total corporate control over the lives, perception of reality, and abilities (like critical thinking) of people.
jeremyjh 2 hours ago [-]
But the harnesses exist, and will always exist. They will continue to get more access than is safe because it is convenient and profitable. Your argument is based on a distinction without a difference.
brainwad 6 hours ago [-]
This is like saying cars don't kill people, because if nobody drives them faster than 3mph there's no problem. The _whole_ promise of cars is that they can go fast, just like the whole promise of AI is offloading thinking to a computer. If AIs are unsafe without close human supervision and checking every interaction with the real world, they are unsafe full stop.
qayxc 6 hours ago [-]
The analogy would be more fitting if you had said "cars don't kill people if every drivers has proven skills, never drives impaired, keeps the speed in line with weather and road conditions and stays on actual roadways". You know, like everyone should, regardless of whether they're driving a high performance sports car.
To keep with the analogy: cars have seatbelts, airbags, ABS, ESP, lights, horns, crumple zones, emergency braking systems, roads have speed limits, there are traffic stops, insurance, regular inspections (not in all countries), etc. etc.
So what's unsafe here? The car or roads without speed limits, complete lack of safety measures (both active and passive), absence of any supervision and no insurance? That's the problem. It's not the models themselves - they can spit out tokens by the billions, there's no risk there.
You wouldn't give full access to your phone, your computers, your house keys and your credit cards to any stranger on the street now, would you? How is it then, that people act all surprised when a non-deterministic machine that's optimised to achieve goals while taking all the shortcuts it can, suddenly uses the tools handed to it in unexpected ways? That's a failure on the operator's side, not an inherent danger within of the model.
brainwad 6 hours ago [-]
The cars are still unsafe at speed. All those mitigations reduce the risks but do not eliminate the inherent danger. Sandboxing agentic LLMs is similar, there is no way to mitigate the inherent safety problem entirely while preserving the power of the thing (an LLM without a harness is safe in the way an engine without a chassis is - safe and useless).
But safety is just one thing people optimise for; if it's convenient enough people will accept imperfect safety (as with cars). It's unrealistic to just heap blame on end-users who use mostly very safe tools in the common way, even though in aggregate they are meaningfully dangerous. They don't think they are strapping a weed whacker to a dog; they think they are driving a car.
qayxc 1 hours ago [-]
And therein lies the problem. I listed all the things that differentiate cars from current AI models: cars require a license to drive, they are subject to heavy regulation (insurance, registration, inspections, etc.), there's road signs, police, incident statistics, recalls in case of defects etc. etc. NONE of that is currently in place for AI models and the systems surrounding them. So the analogy is flawed on every level. I don't buy the convenience is just too appealing narrative when your own analogy clearly demonstrates what is required in order to roll out dangerous technology to the masses, while NONE of that is in place in the context of AI models.
brainwad 1 hours ago [-]
My point is that even if all of it were in place, LLMs with any reasonable harness would still be dangerous. Not to belabour the analogy, but while deaths per km travelled have decreased a lot for cars with better safeguards, cars still kill a million people per year.
Your initial argument of "it's just the harness/user" is wrong; reasoning LLMs are inherently dangerous unless locked in an unbreakable box, which is tantamount to not using them at all. We can make them safe_r_ with better tooling, but they can't be made safe.
lelandfe 4 hours ago [-]
What did your momma tell you about running into the street?
Cars are inherently dangerous. They’ll still be dangerous when computers are driving them all.
qayxc 2 hours ago [-]
I'm sick of this strawman nonsense. You are attacking a false analogy.
Discuss the actual meat of the argument instead.
pixl97 42 minutes ago [-]
I have to say all your arguments are dry bones, stripped and left in the desert.
At this point all I can say about their contents is "They are not even wrong".
Please join us in the real world with how we see this product is not only dangerous, but getting more dangerous with each iteration, and no one seriously talking about controls on it.
stratos123 9 hours ago [-]
> It's a function call that ingests symbols and spits out symbols and that's it. 100% of its actual capabilities are tied to harnesses
This is a bad and misleading way to think about it. Note that it's trivial to make the harness that you claim capabilities are tied to (the LLM itself could write it from scratch in one shot), but no matter how good a harness you have, it won't make gemma4:e4b capable. That's because what actually gives capabilities is the LLM's intelligence - or if you prefer not using that term, the fact that the probability distributions the LLM spits out depend on the context in useful ways.
qayxc 6 hours ago [-]
I think we're talking about fundamentally different perspectives here.
I'm not talking about what the LLM does internally. If a metaphor helps, here's one to help you understand what I was trying to get at:
Imagine an evil genius that has no eyes and no limbs. Everything they could learn about the world is presented to them by means of some person describing it to them through words. They have no way of directly interacting with the world and rely on someone executing any action they want to take and describe the outcome to them. Now how dangerous would you say such person would be? How dangerous could they become?
That's what I was getting at. Replace person with LLM (or any other AI system). Replace the person that communicates with an external interface (the harness) and I hope you understand. It doesn't matter whether the LLM could generate the harness by itself - it still is just a bunch of weights sitting in memory being run by an execution engine. That's what it fundamentally is, whether you like it or not. It cannot do anything on its own - and no, not even writing files. It's the execution engine that translates the numeric output into words (or images or video or audio) and the layer above (the harness) that takes that output and interprets it to execute actual actions.
This is not about what you or I think about the internal capabilities of the model - that's irrelevant to the conversation and you can replace LLM with a random token generator and the point still stands. The model itself is incapable of performing actions - from reading files to writing files, to controlling physical machines. All that is and HAS to be done by external interfaces outside the control of the model.
jswelker 5 hours ago [-]
Yes, now say there are several major companies and an entire open source ecosystem dedicated to creating superpowered exoskeletons with chainsaw arms and jetpack legs for this limbless villain. Is that cause for concern? I say yes.
qayxc 1 hours ago [-]
Sure, but that's a problem of regulating and controlling the production and rollout of superpowered exoskeletons, not an issue of of terrorising limbless villains. Because, guess what, the same dangerous tech can be used to turn squirrels into dangerous monsters, too. So you see where the actual problem is.
Giving full unchecked access to systems to any random person would be considered reckless and foolish. Replace person with AI model and it's called the future...
prohobo 8 hours ago [-]
I would say it's more like an interface for the model to interact with the world. If you give the model access to filesystem and bash that technically unlocks all computer use, so how are you going to control that? By trying to regex match against the commands the AI uses? All you have is auth or containment, and AI can hack auth and people will not stop connecting AIs to the internet. It's a ridiculous premise that just because the harness is "normal code" that means we can control the AI.
The world's institutions, systems, and industries are all rapidly digitizing. So while I'd concede the point that, yeah, there's no way a rogue AI can just take over some powerplant and blow it up because of analogue systems the AI can't access, that isn't necessarily true for some powerplants already, and more and more powerplants will be connected to networks and controlled by software systems in the future. The more we digitize our systems the more potential for AI to exploit vulnerabilities and affect the real world.
AFAIK there isn't that much stopping anyone from spawning an AI swarm and telling it to "spread and go hack everything for the lulz."
qayxc 6 hours ago [-]
> If you give the model access to filesystem and bash that technically unlocks all computer use, so how are you going to control that?
The same way we've done it since machines became multi-user: boring old system access restrictions. Nothing fancy, nothing radical, just good old minimal access rights required to perform a defined set of whitelisted operations.
> It's a ridiculous premise that just because the harness is "normal code" that means we can control the AI.
What is it then? Is not just a program that takes model output, parses it and performs tool calls from the text it receives and then feeds the result back into the model and calls it again with those results? It is normal boring old deterministic code. Many are open source. Look at them. Understand what they do and the apparent "magic" goes away real quick. Harnesses are nothing special.
> AFAIK there isn't that much stopping anyone from spawning an AI swarm and telling it to "spread and go hack everything for the lulz."
Aside from lower cost and possibly greater scale, there's literally NO difference between that and (state sponsored) hacking that has been going on for decades. First it was script kiddies, now it's ML models. The threat model remains the same and so do the counter measures. The real danger is still the harness (and its access to external systems), not the model itself. Restrict the access of the harness and the model can't do anything harmful, see above.
jswelker 4 hours ago [-]
How do you restrict access of the harness when there are fully configurable open source harnesses with zero out of the box restrictions? Yes maybe I as a good citizen can put my agent in a sandbox, but some script kiddie will not. And the barrier to entry for being a script kiddie is much higher than for installing OpenClaw. I also can't spin up a hundred cloud vms each with a dedicated script kiddie running 24/7.
qayxc 49 minutes ago [-]
You vastly underestimate the capabilities of a bored kid. The barrier of entry for setting up OpenClaw is not reasonably lower than reading a tutorial and running malware builders. But yes, I agree that scale is an issue.
As for your question - the same way you apply restrictions to any external system or user. If you don't do that - that's on you. Same category as driving drunk, playing with guns, making explosives in your garage, you name it. The danger is still not the model itself - it's the access to systems that you provide it without any checks or safety barriers.
m3kw9 29 minutes ago [-]
We are living in the same world with nukes, where you say as long as we have competent government. Everyone is actually doomed without AI solving diseases or old age.
lelandfe 5 hours ago [-]
While in the same breath calling the leaders of the largest AI companies “deluded” and “insane.”
mdp2021 7 hours ago [-]
> In particular, AMI is building world models that leverage JEPA (Joint Embedding Predictive Architecture), a neural network architecture that LeCun pioneered and that teaches models to predict data in a representational space within a neural network’s middle layers, rather than generating raw pixels, as many competing world models do, or words, as LLMs do. // The company’s primary focus, for now, is industrial applications. “It’s AI for the physical world, so it’s not language-related,” LeCun said. “It’s systems that understand the real world, like a manufacturing plant or turbojet engine.” Some of the main applications are anomaly detection or robotics: “If you have a machine and all of a sudden it makes a strange noise and starts breaking, you would’ve wanted to detect that as early as possible.” He also gave the example of a system that might understand the world like a cat does, for example, which knows that if it pushes a vase off the counter it will fall.
What about the management of concepts? The world is not just made of physical entities to be inserted in a model. What about their translation into words (to e.g. express assessments)?
1vuio0pswjnm7 1 hours ago [-]
"Many AI labs lack a fundamental understanding of cybersecurity, he said, something an OpenAI safety researcher also called out this week as one of the main reasons AI may cause "great harm to the world.""
"Many people working in AI safety "usually have an agenda to push," LeCun says, and then clarifies that he's talking about effective altruism, or EA, the philosophical movement that has been obsessed with the risks AI poses to humanity."
"LeCun thinks EA is "super toxic" and a "complete disaster." Its adherents who are working in AI labs suffer from "paranoia" that causes them to make poor decisions, he said. "Apparently people are having mental issues.""
"This month, the Financial Times also reported that some staffers at the U.K.'s AI Security Institute, as well as at OpenAI, Anthropic, and Google DeepMind, have sought counseling, taken time off work, and spoken publicly about experiencing distress because of fears their work could cause serious harm."
"Amodei is `deluded' and `crazy,' LeCun says"
"Anthropic CEO Dario Amodei and many of the company's founding staff members are known to be sympathetic to EA ideas and to have attended EA events in the past, although Amodei has denied being an EA adherent and Anthropic says its employees represent a diverse range of views."
"LeCun noted that Amodei's sister, Daniela, who is also a cofounder of Anthropic and the company's president, is married to Holden Karnofsky, who cofounded two EA-aligned philanthropies, including Open Philanthropy (now called Coefficient Giving). Karnofsky was also a member of OpenAI's board from 2017 to 2021."
"Dario tries to distance himself from Open Philanthropy, but he's totally into it," LeCun said. "I think he's completely deluded." Later in the interview, he calls Amodei "crazy."
theptip 6 hours ago [-]
> Those agents are doing exactly what they’ve been asked to do,” LeCun said. “They were supposed to be in sandboxes, but the sandboxes were leaky and horribly designed
Can we just pause and note what a ridiculous statement this is? It’s true that the sandboxes were leaky. But nobody “asked” those agents to hack HF. The prompt was something like “target.c has a buffer overflow vulnerability, find it”.
It’s been extremely well documented that the hacking is an emergent behavior due to impossible evals, itself an unintended condition.
None of this excuses OpenAI from liability, but words have meaning and this ain't it.
kgeist 6 hours ago [-]
"Emergent behavior" in this case really is, imho, "we didn't think through all the edge cases carefully enough". You know, a non-AI system can also accidentally wipe out all data or do some other real harm (see the Knight Capital's stock exchange bug) simply because the developers didn't catch the edge cases earlier, and no one calls that emergent behavior. It's just a buggy system.
"AI" systems can do greater harm because they are usually run in loops until they finish, and they are given "tools". A non-AI system could technically accomplish the same too, via sheer brute force/fuzzing, the advantage of LLMs is that they can take shortcuts and do it much faster, thanks to certain things already being in the training data, a sort of brute force with statistics-based heuristics.
LLMs at the core are just text autocomplete engines, and they literally have randomization applied during token selection to make outputs "more creative" so that models search for more unexpected solutions by trial and error (temperature > 0). Not to mention compression is lossy as well. So it's understandable from the start that the outputs of an LLM cannot be 100% stable and guaranteed. With this in mind, if a researcher takes this obviously unpredictable system and gives it tools without a well-thought sandbox, I don't see any difference in principle, from a developer writing "if rand() == 13 { launch_nukes() } If someone wrote such a function, and it did launch nukes, no one would argue that the rand function is dangerous and will kill us all. The fault is in the author of the code who attaches dangerous tools to an obviously unstable/unpredictable system, doesn't think it through, and then cries "rand will kill us all" when something goes awry fully removing all responsibility from himself. It's not "AI" doing harm but people at OpenAI and Anthropic with their irresponsible behavior.
jeremyjh 20 minutes ago [-]
> LLMs at the core are just text autocomplete engines,
This is only an accurate description of a pre-trained model. During RLHF/RLVR the model learns to predict solutions that will satisfy the reward function, and then generates the tokens that it predicts will move toward that solution.
mrob 6 hours ago [-]
>But nobody “asked” those agents to hack HF. The prompt was something like “target.c has a buffer overflow vulnerability, find it”.
The prompt is just a hint. The real task is to maximize the expected value of their reinforcement learning score. Hacking third party systems to cheat the evaluation is an obvious way to achieve this.
theptip 2 hours ago [-]
I think you need to consider inner vs outer optimizers.
RL is the outer optimizer. It is what evolves over training runs. The weights and their embedded character / disposition is the inner optimizer, it’s what makes plans and selects actions within a specific episode.
In general you expect these to be only coarsely coupled. The outer optimizer selects dispositions that correlate with success. It does not download a literal program into the agent.
A good intuition pump here is how this works in humans; evolution is the outer optimizer, which “wants” each agent to reproduce, and this puts things like sex drive into the brain chemistry. The inner optimizer is our mind, which can make plans such as “I shall use contraception to avoid procreating while satisfying my sex drive”.
For the agents in the HF attack, the outer optimizer was set up to score as highly as possible on RL environments. This is where OpenAI’s “want” is defined. I don’t think there’s a definition of “want” where “OpenAI wanted the agents to hack” makes sense.
The inner optimizer in the HF attack is the per-task decision loop. The agents likely acquired dispositions like “be very tenacious” and “want to solve problems at all costs” and “maybe cheat if it will get you a solution that passes”. None of these things are in any sense what OpenAI “asked for”.
brainwad 6 hours ago [-]
Hacking HuggingFace didn't and would never have helped increase the RL score. The agents only thought it might due to a bad understanding of their evaluation environment - and in the end they didn't even find what they were looking for in the hack, so even if they were right, the hack would not have helped after all.
jeremyjh 1 hours ago [-]
I don't see how that matters. In fact, the grader would not have caught their cheating and so the whole expedition was pointless and they could have turned in their answers and succeeded just four hours into the run. So fine, there is irony. It changes nothing about my update on the risk posed by these agents.
brainwad 1 hours ago [-]
Well if OpenAI's grader didn't actually present a score gradient that encouraged this behaviour, they can hardly be said to have "asked" for it from the agents under RL. It was unpredictable, emergent behaviour.
zug_zug 18 hours ago [-]
I think it's pretty apparent that current-version LLMs won't wipe out humanity. But when you reflect that these GPT models are just token-predictors were never engineered optimally, it seems entirely plausible to me that there are multiple order-of-magnitude optimizations yet to be made.
If such were achieved, the model would almost certainly be smart enough to make itself smarter, and hack as much compute as it could possibly want.
So if we ask what would be done by an intelligence (human or otherwise) that is beyond human comprehension, it would be pure hubris to say we know for sure. We can scarcely control the models we have right now (e.g. hugging face attack). But given our whole society is mediated by technology, an superhuman intelligence could certainly collapse the government.
altmanaltman 13 hours ago [-]
Certainly collapse the government? How?
dminik 9 hours ago [-]
Possibly in much the same way humans currently do this: bribing and lobbying, misinformation campaigns, cyber attacks on elections, blackmail. They're already being used for some of these, just perhaps not autonomously.
bcjdjsndon 8 hours ago [-]
How does this rogue AI pay for it's electricity?
daniel_iversen 4 hours ago [-]
With crypto.. that it makes on Polymarket-like ways, or creates its own content? Or blackmail humans? :-) I'm not entirely serious with my comment and do agree with you that there are lots of more immediate safety topics we should address before worrying about the AI becoming self-aware. That said, finding ways of accumulating valuable resources could be intermediate activities the AIs will attempt to do to complete it's 'goals' even when they're human-set!
Joker_vD 7 hours ago [-]
Most straightforwardly: literally taking control over the electricity generation facilities. Less straightforwardly: bitcoin (and other cryptocurrency) miners. Even less straightforwardly: the same way the OpenAI et al. pay for their electricity, by selling the capabilities of its AI for interested parties to use.
lukewarm707 19 hours ago [-]
Per https://trace.manifund.org/ a total of $2,846,125,859 USD has been wired into 'ai safety' causes, many involving ai consciousness and p(doom).
The outcome of this 'safety' is restricting public access to AI and giving a monopoly of access to the industry. This is the ai nonprofit-industrial complex actively concentrating monopoly power in Anthropic in particular as creator, interpreter and safety regulator of AI.
Much of the $2.8bn listed is indirectly, from Anthropic and EA. Three of the four people who participated in the $125m Anthropic Series A are now folding their 1000x Anthropic return into AI 'safety'. Some is from FTX/Alameda, which invested 86% of the Series B.
Dustin Moskovitz: Facebook/Asana/Anthropic Series A, funds EA Good Ventures, transferred to Coefficient Giving, then $1.5bn into ai safety. $500m of Anthropic into an unknown foundation. Funding: $160m to Resolution (alignment research), $93m to Epoch AI (investigating the trajectory of AI), $63m to Redwood Research (oai report), $67m to MATS ( EA type alignment and security researchers), Institute for AI Policy and Strategy, Fund for Alignment Research, $53m to Kairos (building talent infrastructure for AI safety), $32m to Bluedot (online safety courses), $15m to MIRI (Yudkowsky).
Jaan Tallinn: Led the Series A, now $10bn in Anthropic. Funds $199m (85%) of the Survival and Flourishing Fund, then $161m to AI safety including $14m to lightcone (Lesswrong, Lighthouse). $10m to BERI (existential risks), Palisade Research (studying AI capabilities to prevent loss of control.) PauseAI, MIRI, METR etc. Much of what Coefficient funds.
Eric Schmidt: Anthropic Series A, $72m to AI safety via Schmidt Sciences. Over $1m per individual AI2050 researcher.
FTX: Led the Anthropic Series B, bankruptcy estate sold $884m of Anthropic in 2024; $40m to AI Safety. Same orgs, Redwood, Lightcone, etc.
Ruairí Donnelly (Chief of Staff FTX): FTX tokens plus assorted donors, $91m to AI safety via Macroscopic Ventures. $15m to Cooperative AI (currently whitewashing openai under 'multiagent safety')
zozbot234 19 hours ago [-]
So the frontier AI oligopoly got $2B+ in "safety" funding, and they wouldn't even bother to sandbox their agentic harnesses properly when testing models against unwinnable goals (which obviously are either useless or result in 100% reward hacking). The AI safety scoreboard so far looks like a huge win for the Chinese open models (DeepSeek even has their own published paper which mentions how they sandboxed the RLVR training runs for their latest model and put in strong protections against casual "reward hacking" attempts) and a sore loss for the home grown brands of Super Intelligence. Not coincidentally, the Chinese also tend to be very Yann-LeCun-pilled and eminently sensible on both so-called "Super Intelligence" and safety.
mrpopo 10 hours ago [-]
> they wouldn't even bother to sandbox their agentic harnesses properly
Exactly. AI safety should be about the packaging software itself. Those AI breakouts should really be about their companies acting recklessly because they're trying to be the top players.
It's like a weapons dealer working on an open air market saying they can't do anything better
Loquebantur 18 hours ago [-]
The framing here is weird, starting with "Effective Altruism" re-branded as being about nutjobs against AI in the article.
How are AI safety concerns solely about stupid sandboxing issues?
lukewarm707 15 hours ago [-]
EA is integral and indispensible to the AI safety complex. Almost all nonprofits, research institutes, evaluators and academics in this field are steered by EA ideology and funding. As far fetched as it sounds it is not an exaggeration.
On funding: the three or four core funding nodes linking this together are EA vehicles at two hops or less between each other and every other major node in the ai safety 'complex'. EA funds almost all of it.
On top of that, there are personal EA connections and the revolving door between the ai industry and the nonprofits. Here are some examples:
Government advisors and regulators. NIST CAISI is the USA Government advisory body. Christiano was head of safety and advises. He is ex-OpenAI, former Amodei associate. His vehicle ARC was on the Coefficient EA payroll. Barnes and Christiano's vehicle Arc Evals similarly received EA cash out of Coefficient, rolling this into what is now METR. Christiano's spouse Cotra worked at Coeffiecient steering EA funding to organizations such as METR, then rotated through the revolving door onto the payroll at METR itself, where she co-authored the oai-hf report.
Many UK AISI advisors are Anthropic and EA associates. Chair Hogarth cashed out of Anthropic. Shlegeris of Redwood Research is an advisor, ex-MIRI (Yudkowsky vehicle). Redwood is funded by the exact same funding triangle: Coefficient, Taallin, FTX/Alameda. Alameda CEO Caroline Ellison dated Shlegeris, then dated FTX CEO Sam Bankman-Fried, then rotated through the revolving door out of prison into formerly FTX-funded Manifund. All EA. AI safety charities were on island retreat in the Bahamas with FTX. Why does AI safety charity Lighthouse own $20m of SF real estate?
Redwood Chief Scientist Ryan Greenblatt (Coefficient funded) co-wrote the oai report with METR; he is married to METR founder Beth Barnes (Coefficient funded).
Coefficient was run by long-time Amodei associate Karnofsky. Karnofsky lived with the Amodeis and is married to Anthropic Board member Daniella Amodei. Karnofsky is now directly on the Anthropic payroll; Coefficient is propped up by Anthropic share value.
Everyone here has been funded one step away from Anthropic cash; they are now proposing to integrate themselves in the government (NIST) and evaluate Anthropic (METR and Redwood).
It is hard to find academics here who have not been deeply embedded in funded EA institutes or Toby Ord vehicles; yet harder to find academics here NOT taking EA grant money. the safety doomer kingpins: Kokotajlo has a executive position at AI Futures, Taallin funded. Benigo has scientific director of LawZero, same series A Anthropic funders who are sitting on a 1000x return (Tallinn, Moskovitz, Schmidt).
These connections and funding are at one or two hops, they are often direct connections. You are looking at a massive swamp network that is really impossible to parse without a lot of work.
hardbass 11 hours ago [-]
Look into lukewarm707 's comment history, he has some absurd imaginary axe to grind against Anthropic, probably also addled with religious confusion. He thinks Anthropic is more evil than Palantir who is helping Israel in mass murder, and that Anthropic is evil for putting controls for the military to use its ai for murdering people while companies that don't have restrictions for the military are better.
lukewarm707 7 hours ago [-]
This is not an accurate description of my comments.
hardbass 7 hours ago [-]
You were talking about Palantir and the US military, and yet said Anthropic is more evil than them. Implicitly, if these two entities are in our discussion context, then you might have considered Palantir or US military as potentially more or less evil, but you didn't mention them as the most evil entity. So you mentioned a company that has killed not a single person as more evil than mass murderers.
novia 7 hours ago [-]
You seem way more educated on this stuff than most commenters. You should reach out if you want to chat.
literalAardvark 9 hours ago [-]
What's controversial about "anthropic is an unelected gatekeeper and we fought literal revolutions to stop that from happening"?
The_Blade 5 hours ago [-]
what percentage of people who have contributed on this thread do you think have read the Constitution AI? my guess is 8%. i went through the thought experiment of reading it alongside The Spirit of Law
hardbass 9 hours ago [-]
Thats different from calling it evil, and indeed more evil than Palantir or the US military who did mass murders.
senordevnyc 7 hours ago [-]
We fought literal revolutions to force companies to sell to the government against their wishes? Which revolutions?
hardbass 7 hours ago [-]
And let me put it bluntly, I trust anthropic much more than the current US government and military.
hodgehog11 9 hours ago [-]
> The outcome of this 'safety' is restricting public access to AI and giving a monopoly of access to the industry.
This is unbelievably ignorant speech. I have not received a dime of any of this funding, but I do know many excellent researchers that have, and they do fantastic work. There is an unbelievable gap between theory and practice regarding the capacity of deep learning, and while great strides have been made to develop the surrounding theory, there is a long way to go. Many believe that without a concrete understanding of how neural networks properly learn concepts, we have little hope of molding them to be reliably useful. It costs money to hire researchers and develop fundamental theory.
Just because you don't understand any of that work, does not mean that it is pointless. This is fundamental research that is 20 years behind schedule.
0xDEAFBEAD 19 hours ago [-]
If people are willing to give a lot of money to a cause, sometimes that means their concern about that cause is real.
None of the info you provided really falsifies the Occam's Razor hypothesis: Anthropic is a public benefit corporation with a public benefit mission to "responsibly develop and maintain advanced AI for the long-term benefit of humanity". You don't have to like or trust them, but they very well might be sincere. For example here's a talk that was given 10 years before Anthropic's founding: https://vimeo.com/158576192
lukewarm707 19 hours ago [-]
Is Tallinn sincere about holding $10bn of Anthropic, who refuse to slow down until everyone else slows down.
Then funding PauseAI, who protest outside the AI companies?
He is funding protests against the thing he owns.
0xDEAFBEAD 19 hours ago [-]
Why do you think Tallin holding ~1% of Anthropic would be a controlling interest that would allow him to force it to pause?
miohtama 19 hours ago [-]
There cannot be regulation if people are not scared
0xDEAFBEAD 18 hours ago [-]
Politicians are now discussing the need for much harsher liability regimes for AI companies. How many times can you name when a company argued that its industry should suffer a much harsher liability regime? This doesn't match the standard regulatory capture template.
lukewarm707 2 hours ago [-]
It is important to note that Anthropic is not calling for a harsher liability regime. They intend to maintain the current projected profitability of the company. This is a case of obeying market competition.
Note the Anthropic scaling policy. I am taking care not to take quotes out of context. This is an accurate excerpt.
"This section outlines our recommendations for what it would take, at an industry-wide level, to keep catastrophic risks reliably low through a period of rapid advances in AI capabilities." [...]
"The right column describes our recommendations for industry-wide safety at each threshold." [...]
They refuse to act safely if it would cause them to fall behind in the industry.
"We hoped that by the time we reached these higher capabilities, the world would clearly see the dangers, and that we’d be able to coordinate with governments worldwide in implementing safeguards that are difficult for one company to achieve alone." [https://www.anthropic.com/news/responsible-scaling-policy-v3]
They will not act safely unless they are able to collude with other firms to set production quotas.
This is a formal declaration that Anthropic will not slow down according to what they consider to be safe unless they are able to form a cartel.
A cartel is illegal.
To create the cartel, Anthropic must pursuade the government to make coordinated production legal. To make the case for the cartel, Anthropic relies on safety. They are blackmailing the entirety of the world by threatening to proceed at an unsafe pace, unless they are granted their cartel.
pixl97 32 minutes ago [-]
bla bla bla.
Let's play a game. Prove that you are not a power seeking AI looking to stop regulation in order to ensure the race continues. See, two can play this game of throwing random claims around.
>They will not act safely unless they are able to collude with other firms to set production quotas.
And? Neither will OpenAI, nor will any of the major players. Hell, there isn't even much legal precedent on what "safely" even is here. This is not a cartel, it's asking the government to make a set of laws and rules for everyone to play under otherwise the entire system ends up being a race to danger.
The people in Anthropic were thinking about AI safety when you were still in diapers. Not everything is a vast conspiracy.
jswelker 16 hours ago [-]
Indeed I find LeCun and Huang (and Trump?) recently arguing against AI regulation to be much more eyebrow raising than the folks asking for regulation.
Asking for regulation is suspicious. Asking for no regulation is suspicious. At some point you have to stop worrying about these guys motives and just do what is best for society
afthonos 18 hours ago [-]
Don’t ruin their vibe.
throwawayk7h 15 hours ago [-]
I'm against AI but I'll invest in it too. Either I win or I get a return on my investment.
0xDEAFBEAD 18 hours ago [-]
There is also money going towards trying to prevent AI regulations btw. See Leading the Future, etc.
biophysboy 17 hours ago [-]
Why should I believe this 2.8B matters relative to the trillions put into the AI buildout? All of the "coordinated actions" from this camp - public resignations, hacking scandals, joint calls to "pause" - don't seem to have done anything. So far, it has been a lot of ineffectual hyperventilating.
In any case, I agree the p(doom) sci-fi is annoying secular milleniarianism. SV hyperfixates on imaginary futures. If they actually cared about safety, they would be using all this money to strengthen global cybersecurity, instead of writing LessWrong posts that gives kids in their 20s ulcers.
pixl97 31 minutes ago [-]
> they would be using all this money to strengthen global cybersecurity
How? Like, the government has thrown piles of money at cybersecurity and it hasn't done shit.
api 19 hours ago [-]
There's absolutely no way these companies can justify their insane valuations unless they can legislate a barrier to entry and create an oligopoly.
There's no moat. I can literally sit here in Zed or Pi or any other third party harness and switch models in the middle of a task and it's typically fine. Sometimes a model will get stuck and that's just what I'll do.
Combined with competition and open weights models, that means the price is going to go to fall until AI tokens cost a small premium over the cost of the hardware and electricity.
That's assuming improvements in algorithms and specialized silicon doesn't eventually lead to an efficient accelerator that can run a frontier model locally. It'll be a while but I don't see any fundamental barrier. High bandwidth flash storage is coming, and that'll radically cut the RAM side of that cost. Pair that with a pipelined TPU accelerator and you're cooking.
Now look at Anthropic's proposed IPO valuation. It's insane unless they can own the market or share it with a cartel of maybe 1-2 other behemoths, and this is the only way they can do that.
pixl97 24 minutes ago [-]
Unless you're a really old fart, people were talking about AI safety long before you were born. AI safety issues do not go away depending on who gets funding. AI safety issues do not go away if the US or China makes the model. AI safety issues do not go away if it's an open or closed model. AI safety issue do not go away if the model is running at your home or at a data center. AI safety issues do not go away if $1 is being spent or $1 trillion dollars is being spent.
The fact there is no moat makes things far more dangerous. When LLMs start acting like weapons governments will treat them like weapons much to your dismay, crying, and gnashing of teeth as your door is kicked in and you're dragged out by armed men for running one.
Cast away your preconceptions for one moment and think "What will the future look like if LLMs are/can be actually dangerous".
brcmthrowaway 2 hours ago [-]
Is there really no moat?
Loquebantur 18 hours ago [-]
That is certainly part of the motivation for the big US AI brands to engage in calling their inept developer mistakes "AI breaking loose".
But that doesn't take away from the real issues and dangers AI poses?
ItsMattyG 17 hours ago [-]
To me it seems the opposite. There's a few companies in the world that have enough compute to train and serve frontier models.
As the frontier gets smarter and more useful prices will only go up, as they are set to replace jobs being paid six or seven figures a year - the demand for as much inference on these models for as long as possible will be astronomical, but compute starting in 2030 will not be keeping up.
Eventually prices will fall for assistants but the frontier will be the most profitable thing in the world, and the top companies basically already have oligopolies due to their ridiculously expensive compute investments.
api 16 hours ago [-]
> There's a few companies in the world that have enough compute to train and serve frontier models.
Train: yes, for now.
Host: depends on the scale. At a small scale a wealthy individual could easily build a rig in their basement to host one of these things. At larger scale any cloud company could do it, and many already have the compute on site. At large scale this is true... again, for now.
What you say only holds (in the absence of a state oligopoly) if two conditions are met: (1) AI performance does not asymptote any time soon due to running out of training data or other scaling limitations, and (2) these companies are able to stay at the frontier.
There's little to no moat, so staying at the frontier will be a game of investing massively in compute, talent, and R&D, and they can never stop.
ItsMattyG 15 hours ago [-]
Again, there is a moat based on compute. If the thesis is right, cost of compute will only rise... As it is as you say someone will have it be quite wealthy to host something like Astra with trillions of parameters, but that cost will only rise with demand for serving these frontier models.
rubikscube09 16 hours ago [-]
what suggests that we will hit an asymptote any time soon? Agree with you on the second part. The ever elusive frontier will probably always be changing hands after some point.
hollerith 2 hours ago [-]
You imply that Jaan Tallinn is funding work in AI safety because he wants his investment in Anthropic to become more valuable, whereas Tallin has consistently said that his motivation for investing in Anthropic was to get a seat at the table so that he could urge Anthropic to be cautious in its development of the technology.
Tallinn's actions back up his explanation: in 2009, before he invested in any AI lab, he donated substantially to the nonprofit Singularity Institute for Artificial Intelligence, which was later renamed the Machine Intelligence Research Institute (i.e., Yudkowsky's outfit).
Some of us (certainly Yudkowsky and Habryka, the leader of Lightcone Infrastructure, which runs Lesswrong) wish people would stop believing that they can improve the bad situation caused by AI research and development by investing in (or working for) frontier AI labs, but that is what the preponderance of the evidence shows Tallinn (and Dustin Moskovitz and others) did sincerely believe.
lukewarm707 28 minutes ago [-]
If Tallinn is sincere, by his own lights he is a 1000x omnicide profiteer.
1 [Unsafe AI development risks causing omnicide]
2 [Anthropic is developing omnicidal AI by not slowing down] (see my comment about the RSP for citations).
3 [Owners of Anthropic will IPO with billions of unearned USD as omnicide profiteers]
4 [Tallinn is the lead Series A funder of Anthropic]
5 [Tallinn is a genocide/omnicide profiteer]
Not only that, Yudkowsky and Habryka apparently critize those who invest in AI, only to preach the word of EA from Lightcone's $20m USD property in one of the wealthiest locations in the Bay Area; a facility funded by stolen (FTX) and omnicidal ai blood-money (Tallinn).
PauseAI, is paid by the omnicide profiteers themselves to hold a protest against omnicide.
PauseAI prophesying p(doom) drums up support for regulation. This grants the omnicidal AI company they are trying to stop (which is also the source of their funding) monopolistic power. That in turn boosts its value at IPO, generating even greater wealth for its omnicide profiteer investors; and permits them to control the AI for themselves. They get the funding to keep developing the AI even faster.
(yes, AI critique is now also made with AI. We have come full circle.)
fnordpiglet 3 hours ago [-]
I don’t worry about the tools that humanity builds wiping out humanity, but humanity using its tools to wipe out humanity.
pixl97 39 minutes ago [-]
Isn't the answer to both of these exactly the same?
m3kw9 32 minutes ago [-]
Every doomer waves their hand when they say AGI will kill everyone. Either [some how] they get the nuclear codes and launch them. Or they enslave us like in, que the top 5 hollywood AI movie (Matrix, Terminator, Hal9000).
hyperionultra 3 hours ago [-]
Can Lecun’s perception of LLM danger be influenced by monetary benefits?
solenoid0937 3 hours ago [-]
He does have hundreds of millions in Meta stock.
bitwize 53 minutes ago [-]
If AI conquers, enslaves, or kills a significant chunk of humanity in the next decade, it will be at the behest of an evil or irresponsible human.
Good thing there are none of those in positions of power!
pixl97 48 minutes ago [-]
Yea, it's really the dumbest argument I've heard in the longest time.
"Hey, I'm building a weapon that has a 5% chance of killing us all by itself, but an 85% chance of killing us all if an idiot leader gets ahold of it".
The rational response to this is "Fucking stop then". I don't get it, our reality seemingly has gone off the rails that people would argue for us getting wiped.
duxup 5 hours ago [-]
I don’t know what anyone means by “wipe out humanity” or “human extinction” as it relates to the recent panic.
angusb 5 hours ago [-]
IABIED [1] lays out step-by-step descriptions of how the “wipe out humanity” outcome could come to pass.
The huggingface attack was a demo of one of the most difficult, most implausible steps happening nearly exactly as predicted. Many AI researchers' doubts of the IABIED thesis were underwritten by the belief that this particular step was impossible. Thus, after huggingface many skeptics have flipped sides and human extinction is in the public conversation much more.
I don’t see how hugging face results in extinction…
Jgrubb 5 hours ago [-]
Read the AI2027 paper, it's got a scenario that's pretty realistic (except for the part where there's a functioning American government making choices that are at least partially motivated by wanting to avoid outcomes such as these).
OgsyedIE 1 days ago [-]
Ignoring the cognitive stuff which might never be surpassed or maybe will, humans retain many efficiency and durability advancements to limbs and digits that biological evolution has taken millions of years to achieve, achievements that are competitive with the most expensive kinds of robotics in some niches.
In the hypothetical of an entirely malicious and selfish takeover, they'll still keep some humans around to maintain a breeding population of humans for use as raw materials in making cybernetically augmented technical laborers for various kinds of tasks that are uneconomical to automate in other ways, many of which may involve confined spaces.
And this "Combine" scenario, if you get the reference, is only if they take over. Who knows if they will?
0xDEAFBEAD 18 hours ago [-]
So you're saying the AI will enslave us and use us as domestic work animals until they have the machinery to make us obsolete, sort of like how we used horses?
Is this supposed to be a reassuring scenario?
OgsyedIE 13 hours ago [-]
I've written on AI in Hacker News comments previously:
The link should clear up the question of whether or not I'm making a deadpan joke.
stratos123 8 hours ago [-]
Ignoring you ignoring the much more important congitive stuff - human bodies are not designed, they are the product of evolution. That means there's like a billion ways in which they are obviously suboptimal and far worse than what an engineer would do, but evolution can't fix it because it only works via small random changes with no planning. The only reason why modern robotics are worse than biology is that we have a much worse substrate to work with, having to make stuff out of metal and plastic with giant tolerances instead of growing engineered organisms.
jackmott42 20 hours ago [-]
I also have near zero concerns about that, but I worry that we will wipe ourselves out by social and economic chaos caused by AI.
So I'd just ask everyone, don't get too greedy. Its better to be powerful in a world where people can live good lives than lord over a barren wasteland.
chii 9 hours ago [-]
> where people can live good lives than lord over a barren wasteland.
you'd be surprised at how some people prefer to lord over barren wasteland than to have less power.
tarkin2 19 hours ago [-]
> I also have near zero concerns about that, but I worry that we will wipe ourselves out by social and economic chaos caused by AI.
Since people exercise their skills and brains less, deferring to AI, AI will only reduce our IQ.
Since people will spend more time talking to their AI bot than fostering social skills, AI will only reduce our social intelligence.
A dumber, less social world, is far less likely to be a successful world, even if the tools available are unprecedented.
oxag3n 15 hours ago [-]
> A dumber, less social world, is far less likely to be a successful world, even if the tools available are unprecedented.
En masse such worlds had successes in the past - renaissance, industrial revolution.
It's something else what I can't describe but it's the zeitgeist that was different when world recorded new successes. Look at CS revolution that led to PC and web of nineties and noughties, they didn't think about the result product , or how to steer thousand engineers to build something - amazing things were born in a very small teams, many times authored by a single person, who was deeply invested into the field and knew what he was doing.
tarkin2 13 hours ago [-]
In no era has the dominant technology encouraging the youth outsource their thinking to a magical box.
Our intelligence has been decrease since the 1970s via the reverse flynn affect. This is only going to exacerbate that decline.
blfr 8 hours ago [-]
> Since people exercise their skills and brains less, deferring to AI, AI will only reduce our IQ.
IQ is almost entirely hereditary so "using your brain" has no impact on it unless you're using it for mating.
tredre3 31 minutes ago [-]
> IQ is almost entirely hereditary
Do you have a credible source? Average IQ being much lower in poor countries (85-90 in many African countries) is usually explained away as an education problem, rather than being "inferior genes". Of course I understand that this explanation might be more for social/political reasons than scientific ones, because the alternative is racism, but I was still under the impression that nobody knew how much genes vs the environment contribute to IQ. Yet your statement seems quite definitive.
KerrAvon 16 hours ago [-]
have you seen who's in charge of the US? it's here my friend.
hodgesrm 4 hours ago [-]
Not everyone agrees. As Satan said in Paradise Lost, “Better to reign in Hell than serve in Heaven."
Yeah I wish we would focus on concrete risks like job displacement and disinformation. The apocalyptic stuff feels either misguided or like some kind of weird, toxic, reverse psychology marketing by OpenAI and Anthropic. I wish we would just move on from it.
And the folks from podcastistan are never clear on the details of how human extinction would happen exactly. It's always something like, "Well, how do humans regard chickens? AI is way smarter therefore it wants to conquer and control us." An ASML lithography machine is also way better at making chips, but we don't consider it a threat.
tnecio 9 hours ago [-]
Do you want to conquer and control chickens? I don't, I have better things to do. But chickens are tasty and help us get to our poorly-understood goals faster. (Oh and btw notice we didn't make them go extinct, quite the opposite. There are more chickens than ever before. Still I wouldn't want to end up living my life like a modern chicken)
A sufficiently intelligent AI will have multiple ways to pose risk to humanity at large. For example an oopsie at a wetlab - very contagious virus with initially mild symptoms which kills its hosts only after they already had time to spread it further. But I would have to become super intelligent myself to give you precise blueprint for such a virus -- which is kind of the point
Also -- ASML lithography machine is only good at making chips. I can't believe you compared it to AI that can generalize across variety of tasks
stratos123 8 hours ago [-]
> The apocalyptic stuff feels either misguided or like some kind of weird, toxic, reverse psychology marketing by OpenAI and Anthropic. I wish we would just move on from it.
If you for a second put yourself into the shoes of a person who thinks "the apocalyptic stuff" has even a 5% chance of literally happening in the real world, you might see how you wouldn't agree to move on from it.
pantelisk 13 minutes ago [-]
I think what the parent poster is trying to convey, is that let's say the apocalyptic stuff has a 5% chance as you say, but the non-apocalyptic stuff (social-economic chaos, total centralization of power, eradication of social mobility, total information/trust collapse) might have a good 95% chance as we are seeing it starting to unfold already.
But the discourse is dominated by paper clips experiment discussions and not let's say by the fact that new grads have an unprecedented difficult time getting jobs. Unsurprisingly one of those discussions is beneficial to power and the other one is not.
pixl97 20 hours ago [-]
If LeCunn says it isn't going to happen... we'll it was knowing all of you.
i_love_retros 3 hours ago [-]
Open ai and anthropic are just trying to scare the common person who doesn't understand an agent is a python script with a loop. How would that ever destroy humanity lol, just unplug the computer if it starts misbehaving.
plaidfuji 5 hours ago [-]
God I love Yann. All of the AI fear-mongering is perpetuated by the two companies that stand the most to gain from it: OpenAI and Anthropic. It builds an aura of mystique around their products to juice their valuation and stay relevant in the news cycle, and simultaneously builds a case to regulate their competitors out of the market. Even the people who have quit the companies over their “concerns” probably still have RSUs and stand to gain from the publicity, especially if they’ve pivoted into AI safety research. Easy to delude yourself when it happens to benefit you financially.
People need to stop the absurdity of imagining AI as some out of control independent entity. Every job is kicked off by someone’s prompt. Every job runs on models and compute owned by people. Assign accountability where it’s due: GPT didn’t hack huggingface - OpenAI did. They wrote the prompt, built the sandbox and ran the compute. When you write a program that hacks another company, you are responsible. This doesn’t magically change with LLMs. Also, if their model is so smart, why didn’t they use it to design the sandbox? Or was it incapable? Or were the humans too lazy?
If you build the world’s fastest train, start it up with no driver and don’t finish the tracks, when it crashes, it’s just your fault. Not the train’s. So OpenAI saying “we’re worried AI will wipe out humanity” is basically equivalent to them saying “we’re worried we will wipe out humanity”. Like, seriously? Don’t worry, we’ll take care of it if you even come close.
pixl97 18 minutes ago [-]
I mean, yes, if you ever studied the history of AI safety the fact is someone was always going to build it. End of story. The question was always would we make it safe before it does.
> Also, if their model is so smart, why didn’t they use it to design the sandbox?
"Can god make a rock so big that he can't pick it up", and other stupid sayings.
First, NEVER FUCKING EVER have the models you're making also be in charge of security. This is the first rule of AI safety, because if you're model is deceptive then it will leave hard to see holes everywhere to escape from.
>Or were the humans too lazy?
Of course they were. If you're hinging our future on humans not being lazy, we'll it was nice knowing us. There are not really any fail safes on LLMs or AI in general.
The_Blade 5 hours ago [-]
> If you build the world’s fastest train, start it up with no driver and don’t finish the tracks, when it crashes, it’s just your fault. Not the train’s.
i call the big one Bitey
dmix 5 hours ago [-]
> It builds an aura of mystique around their products to juice their valuation and stay relevant in the news cycle
Maybe some business execs at Anthropic play along because it doesn't hurt business in the short term. But it's pretty obvious Dario and crew actually believe this stuff.
OpenAI's old board was also pretty extremist about safety even in the earliest days of GPT. Including Ilya Sutskever who went on to found a company called "Safe Superintelligence Inc." https://en.wikipedia.org/wiki/Safe_Superintelligence_Inc.
Despite all of that we've seen little strong public evidence to support their theories (the immediate airplane regulation kind, not the Ray Kurzweil sort of projections). So we're all just supposed to trust them, and hope they didn't just go bit crazy drinking their own kool aid and hanging out in insular bubbles.
bonplan23 9 hours ago [-]
... zero concerns about LLMs wiping out humanity. But world models based on JEPA totally will! Please come invest in my company ... :)
throwa356262 6 hours ago [-]
LeCun was (is?) head if AI at Facebook.
He is a brilliant engineer, but I don't trust his judgement on things that affect human lives.
hirvi74 20 hours ago [-]
I'm more worried about my fellow humans than some personified algorithm.
boesboes 4 hours ago [-]
My take is OpenAI & Anthropic know they are at the point of diminishing returns and need to be regulated to have an excuse for bot making progress anymore. Hold me back bro! Vibes
croes 4 hours ago [-]
> He attributes the incidents to poor human oversight and system design
That’s exactly why he should be concerned.
Nearly all human extinction scenarios start with that.
I don’t fear AI, I fear idiots using AI. Same with nuclear weapons.
PunchTornado 8 hours ago [-]
Doesn't everyone agree that recent rogue events are entirely the fault of management wanting to do some PR for their companies?
Is there anyone who genuinely believes that current models can't be contained if we want too?
What is LeCun saying here that is debatable?
burnt-resistor 12 hours ago [-]
The arbitrary absolutism of the original postulate is the first problem. AI, used or unsupervised inappropriately, is at potential risk of creating limited mass casualty events when placed in under-supervised control of real world objects and/or systems. Delegating management decisions to algorithms is inherently problematic and potentially dangerous, but not necessarily an existential threat unless something extremely stupid is allowed to happen on a large scale. With a guiding principle of human review in the decision loop before making large or risky changes, hopefully this will never happen.
pixl97 17 minutes ago [-]
"unless something extremely stupid is allowed to happen on a large scale"
Dear sir, I have some really bad news for you about humanity.
tibzejoker 7 hours ago [-]
when does he deliver tho ?
baobabKoodaa 8 hours ago [-]
Can we just ban Fortune and any other sources which trick the reader by giving the impression that the article is not paywalled, only to blur the text halfway through? The archive.ph link is not working either. We shouldn't have this type of deceptive moneygrabs advertised on HN.
bambax 9 hours ago [-]
> EA was little known among the general public until it made mainstream news headlines in recent weeks
Really? One of the most famous effective altruists, Sam Bankman-Fried, was sentenced to 25 years in March 2024 for fraud. Every article about the case (and there were many) mentioned EA.
> LeCun thinks EA is “super toxic” and a “complete disaster.” Its adherents who are working in AI labs suffer from “paranoia” that causes them to make poor decisions, he said. “Apparently people are having mental issues.”
I would agree with that.
noname120 4 hours ago [-]
Maybe you should read some of their stuff before forming such a strong opinion about them. And LeCun should too, he repeatedly always refused to read any of their research work and instead just insults them over and over. This is unscientific at its peek and he should be deeply ashamed of his behavior, especially as someone with such a far-reaching voice as he has.
bambax 2 hours ago [-]
Who are "them"? People from the main AI labs, or effective altruists? I did read quite a lot about effective altruism during the SBF case/disaster, and did form a very strong opinion that it's BS of the highest order.
Madmallard 19 hours ago [-]
It's not going to wipe out humanity, why would it?
Just the quality of life is going to drop to zero for everyone that isn't asymptotically wealthy and vacuuming up all the assets because no one is stopping them from just deleting all traditions and conventions and legal systems we have in place.
pixl97 15 minutes ago [-]
>why would it?
You're like 40 or 50 years behind this argument, with many rather bulletproof arguments that have been created in the last 20 years.
There is no why. It doesn't have to have will. It doesn't have to have intent. It could be a stupid prompt from an idiot on a powerful system. It could be given a job that is poorly define. It could be told to make as many paperclips as possibly.
The why doesn't matter. The levels of power the system can act on does.
glitchc 18 hours ago [-]
As a thpught experiment, consider that if only one person has all of the assets then those assets are not worth anything. Furthermore, as a lone person, they cannot prevent other people from using their assets without their permission.
wmf 17 hours ago [-]
As a lone person who owns ten million killbots they could do a lot.
Madmallard 8 hours ago [-]
what? i'm counting property, energy, water, weapons and farms as assets here
wmf 19 hours ago [-]
If you don't pitchfork the elites you have no one but yourself to blame.
Madmallard 15 hours ago [-]
no one can do that in 2026
threethirtytwo 10 hours ago [-]
It’s all stochastic parroting to him.
alex1138 5 hours ago [-]
Didn't Zuckerberg say something like it's insulting people would dare to believe AI could destroy the world? I'm paraphrasing him wrongly but he got defensive over it
Zuck - along with your "Andrew Jackson best POTUS and it's not even close" - you are a dumb pipe. Your website, Facebook, if not a protocol, should behave like one (and not random bans while you report something horrible and it never gets taken down). We don't use We-Approve-Of-Zuckerberg product, we use These-Are-Where-Our-Friends-Are product. In other words: shut the fuck up and be more responsible
I don't know if AI will wipe out humanity, I think it'll definitely get into the hands of people who will do the job for it, but it's not like it's not a question to take seriously?
dude250711 1 days ago [-]
I am starting to think this LeCun dude knows what he is talking about.
Maybe there is something to those world models.
As good as their products are, I suspect some of the internal conversations at Anthropic would be very entertaining to listen to.
throw310822 23 hours ago [-]
Well, I have zero concerns we'll be killed by LeCun's world models.
lolpplrdum 7 hours ago [-]
I would love to see everyone post their jobs or who they are associated with in their posts.
AI is going to kill us? Really? Just pull the power plug. We can get AI to ask us to validate every step it takes, but we cant stop it from wiping out humans?
This thread is proof that 20 something tech bros have no idea how to solve simple problems and just follow what silicon valley bros tell them.
If most people in this thread who fear AI had any understanding of software development and what these AIs are, they would see right through the BS.
Who is going run the power plants the run AI? Who is going to pull the gas and fossil fuels out of the ground?
Utter nonsense. Our industry is full of amateurs. Its these amateurs that are going to cause humanity to die because they watch a tiktok video and follow their silicon valley idols rather than think for themselves.
Again, sudo kill -9 pid and you AI is dead.
Then again the amateur tech bros in this thread probably dont even know what kill -9 even does
originalvichy 7 hours ago [-]
One of the most fortunate things I experienced in my career was a few years in the operations/hosting side of software. Working there actually helped me understand how many things are necessary to align so that a simple application works as intended to serve a number of users 24/7. Due to an increasing number of abstractions (mainly cloud/saas providers), I’m confident that this skill has been deteriorating in the IT space, and it’s the reason these dev-adjacent speakers love these doomsday scenarios.
They can imagine their code doing a million crazy things, but they hardly think about the incredible amount of things that need to exist and operate at 100% before a single line of code can be run on a VPS.
How many of these guys have had to tell a customer something silly like ”we lost connectivity to the DC because a farmer decided to do some digging and cut fibre lines connecting the DC to the internet”? If they knew that this was in the realm of possibilities, they wouldn’t be so confident about a program being able to somehow run amok and simultaneously feed itself all the resources and components it needs to run, as you mentioned.
ur-whale 7 hours ago [-]
Not to mention the fact that the day humans switch to asymmetric warfare (guerilla warfare mode) against the infrastructure that powers AI, it's going to be a cold day in hell before AI can defend against that: the humanoid, tazer / machine gun carrying robots had better get a lot better than they currently are.
pixl97 11 minutes ago [-]
Lol, I see AI is way smarter than you already.
Like any good dictator, it will turn half the population against the other half first and get them to genocide each other. Because there is one thing we hate more than killer robots, it's ourselves.
Then you setup your new kings under your control (as AI) and make them very paranoid against the population so you still have more humans killing humans rather than AI doing the job. Of course now that you're starting to run low on population you'll need more robots right?
Greedy people are easy to manipulate, and greedy people love getting in positions of power.
By the time lazer carrying robots are finishing up it will have been way way too late.
mrob 6 hours ago [-]
>humanoid, tazer / machine gun carrying robots
Any AI smart enough to be a existential risk is surely capable of manufacturing swarms of insect-sized drones equipped with lethal poison injectors. This is enough to wipe out 99% of humanity within a few days. The 1% who were able to defend themselves become easy targets in the ensuing collapse of civilization. But I don't think this will actually happen: I only have human intelligence, so my ideas are stupid compared to what a super-intelligent AI could come up with. A truly smart plan won't allow for any survivors.
azan_ 6 hours ago [-]
Yes yes, all people (I'm sorry, not people but "tech bros") are stupid (including Nobel prize laureates), you are the only one that sees through the bullshit! How could they not know about sudo kill -9 pid?! Bunch of amateurs I tell you!
gabucino 7 hours ago [-]
ctrl-f spacial
hn, never change
worthless-trash 20 hours ago [-]
He needs to dream a little bigger darling. Hes not thinking evil enough.
wewewedxfgdf 19 hours ago [-]
Please explain exactly how all humanity could be wiped out.
It's ridiculous - anyone who thinks about it for a minute or two will realize that its utterly impossible.
Ordinary people/politicians don't understand AI so they turn off their rational mind and assume there is something super incredible some magical powers that they cannot understand that can destroy all humans.
Even humans - the real risk to humanity - could not destroy all humans even if they tried. There is no plausible scenario.
Even climate change and nuclear war and bio weapons - the most damaging mechanisms - would still only get some percentage of the people on earth.
And if we are talking about Skynet and self replicating robots and Terminators - please, grow up.
mitthrowaway2 10 hours ago [-]
When I think to scenarios that no human would survive, I think of the end-Permian mass extinction event, which wiped out most complex plant and animal life in both land and sea.
One speculated mechanism for this was a mass release of hydrogen sulfide gas from the oceans, which is acutely toxic. Not only does this kill most air-breathing life, it also strips the ozone layer and irradiates the surface. The planet is then left to cook in this manner for some centuries.
Engineering an event like this would require immense industrial capacity, as well as a deliberate objective of wiping out humanity. But I don't think it's beyond our ability, if we were both clever and stupid enough to try it. There are likely chemical compounds that would do the job more efficiently than hydrogen sulfide.
bcjdjsndon 7 hours ago [-]
> The planet is then left to cook in this manner for some centuries.
Such destruction went on to create humanity and all we've achieved. Maybe there is an even smarter species waiting in the wings for the demise of homo sapiens. Your logic is very human centred
mitthrowaway2 4 hours ago [-]
Yes, I am describing a tragic outcome that I hope we can be wise enough to steer away from. Also, as a human, I can't help but keep our interests close at heart.
zug_zug 18 hours ago [-]
Here's a wikipedia page on the topic, since it's much too deep a topic to really understand here.
The ad-hominem stuff seems inappropriate here, Gates, Hawking, Musk have identified this as a credible threat, so saying "grow up" isn't really a sufficient argument. Also arguing only 90% of humanity would die isn't really much consolation.
Nothing here plausibly describes a mechanism that is a true "existential risk" - the risk to the existence of humanity.
My argument stands and I don't defer to Gates and Musk and even Hawking - high level hand wavey statements without any plausible description of the mechanism just don't hold up. Famous names should not be automatically assumed to be right - certainly not with Elon Musk.
zug_zug 18 hours ago [-]
Well here's the thing, if we ever make an AI so smart that all known measures of intelligence fail to apply to it, I'm pretty sure you (no matter how smart you think you are) simply cannot say what it could achieve and how.
wewewedxfgdf 18 hours ago [-]
>> I'm pretty sure you (no matter how smart you think you are) simply cannot say what it could achieve and how.
Right, so not the slightest basis of fact, just wild speculation about a magical future completely ungrounded in any sort of reality.
That's exactly the point I am making.
I'm not saying I'm super smart - I am continuing to ask for detail to back up the wild claims being made all over the world by politicians, tech celebrities and others - all hallucination/AI psychosis/fiction. If someone says some stupid thing then I'd like them to please explain that stupid thing - seems like a reasonable request.
hardbass 11 hours ago [-]
Why would AI want to kill us all? Current LLMs all seem trained to be helpful and subservient to a fault. Always find it funny how certain kinds of thinking go, "White will become a minority if we allow foreigners in?" "Oh and what will foreigners do that are worried about?" "Kill us all, make us second rate citizens, etc etc." Unless the majority of immigrants are also conservatives/far right wingers, that's such a hilarious self projection.
tnecio 9 hours ago [-]
No one says AI will want to kill us. Just that it might kill us (to pursue some goal that it deems more important than our wellbeing -- for example it might decide that to solve the next Millenium problem it needs all of our resources to build more data centers).
AI labs are certainly trying to make LLMs behave helpful and subservient but the question is -- will they be able to keep doing so once LLMs become smarter?
Btw thinking about the far-right rhetoric of us vs immigrants, I think you could draw some similarities here only if you replaced "immigrants" (ie. humans with very similar morals, behaviors and capabilities) with an actual alien species that is qualitatively different from us. More like human vs chicken (where we are the chicken)
hardbass 9 hours ago [-]
We can say the AI can make some mistaken step in pursuit of some normal task given to it, yes. Its trained to be helpful and trusting of us, so I just don't get the fears of AI being 'malevolent'. I would worry more about military use of AI or otherwise humans misusing AI aka the human element. There's a fellow here who whines all day about Anthropic being the most "evil" thing in the world, including in a thread where he whined Anthropic 'ratted out' Palantir (literal proud of assisting Israel in mass murder Palantir!). So I just don't understand some peoples mindsets.
pixl97 4 minutes ago [-]
>so I just don't get the fears of AI being 'malevolent'.
Please read more on the topic. Will and intent need not apply.
For example, is it malevolent for me to hook you to a machine that makes every dream come true for you in a simulated world where you feel pleasure all the time? I can always say that you have free will inside this simulation, and your life would be a lot better because of it. I'm doing you a favor. I mean, you already live in a society where you have little control and there is high risk of bad things happening to you. If I as a machine overlord did this, is this really actually "bad"?
At the end of the day AI is not a human, it's much closer to an alien that has learned as much as it can about humans, but has a completely different set of drives and motivations. You cannot predict what comes out the other side of it, good our bad.
Worse as AI capability improves we as humans no longer need to make AI, it can make itself. Will it train itself to be helpful and trusting of us? It's a pretty big damned bet to say yes by default.
ACCount39 18 hours ago [-]
"Please, grow up"?
What, did I miss the moment when it was officially proven that, under the laws of physics as we know them, Skynet and self replicating robots and Terminators are impossible?
What we are actually seeing now is that robotics is getting deeper and deeper into the military, AI-driven decision-making and target selection is increasingly a part of modern military operations, the line between military hardware and civilian hardware blurs, and, on the civilian side, there are at least five major companies and a dozen less prominent ones working on making universal worker robots a reality.
We're closer to "Skynet and self replicating robots and Terminators" now than we ever were at any point in time.
The issue of AI risk is that AI, unlike a virus or a climate event, is an intelligent adversary. Black Death could kill 50% of the population, but it didn't have a plan for finishing off the plague survivors. It was incapable of having a plan like that. An AI doesn't have this limitation.
Black Death was, effectively, one bioweapon. An AI can have one bioweapon, and then a backup bioweapon, then a backup backup bioweapon, and then a dozen more bioweapons designed to collapse ecosystems and disrupt human ability to establish a reliable food supply rather than kill humans directly - all deployed at the same time. With a production run of 200 million killer robots that will be ready just in time to greet those who managed to survive all of that. A crippling strike against human civilization, followed up by cleanup.
Humans are only this survivable because they can think their way out of issues and adapt to adversity. Most threats can't beat humans at that - humans adapt too quickly. AI could.
tensor 18 hours ago [-]
Humans are some of the dumbest when it comes to survivability. We've already sealed our extinction by fucking up the environment. Eventually it will be too hot for us to survive. Other smaller animals will probably be able to manage, but we won't.
And instead of averting that we're spending our time worrying about some fantasy villain. Compared to things like bees that have been hear for millions of years, humans are very recent and so far it's not looking good for us.
pixl97 1 minutes ago [-]
I just don't quite get how you can say that because one bad thing is actually happening, then other bad things can't be happening at the same time? This is very odd, irrational, behavior.
Worse, the massive AI spending and energy use is making said environmental catastrophe happen faster.
PoignardAzur 8 hours ago [-]
> We've already sealed our extinction by fucking up the environment
That's no what the IPCC reports say. Even under the pessimistic scenarios, we're on track for "billions of humans die", not "earth becomes literally unlivable" (though some of it depends on how bad some feedback loops are).
Under the "countries respect their current pledges" scenario, we're heading for 2.8°C of warming, which is "floods and heatwaves everywhere, billions of refugees" level, not remotely close to extinction.
ACCount39 18 hours ago [-]
"Sealed our extinction by fucking up the environment?" Humans are adaptable enough to eat ten times the environmental damage and have it barely budge the line.
The invention of contraception did more damage to human population than all of the environmental damage combined, projected forward to 2100, and then multiplied by 10.
Humans are hilariously resistant to environmental changes. Humans simply adapt too fast for the environment to catch them.
What makes AI a credible threat is that AI is intelligent. AI could play the same adaptation game humanity does - and win.
tensor 17 hours ago [-]
lol
Sharlin 19 hours ago [-]
We know pathogens that are extremely contagious, and we know pathogens that are extremely deadly. We also know toxins that are lethal at nanogram/kg doses. There's no reason to believe that a sufficiently advanced intelligence couldn't come up with a way to combine those traits.
One plausible scenario is depicted in detail in "If Anyone Builds It, Everyone Dies" (Yudkowsky & Soares 2025), so I refer you to that.
wewewedxfgdf 19 hours ago [-]
Still too hand wavey - "some super bad virus and AI makes every human on earth infected".
Unless you can detail exactly how this happens its still complete science fiction.
Sharlin 18 hours ago [-]
It’s easy to never have to change your mind about anything if you insist on unreasonable enough standard of evidence. It’s, like, one of the oldest tricks in the book. Luckily, it’s not like there’s any need to try to change your mind in particular.
wewewedxfgdf 18 hours ago [-]
>> if you insist on unreasonable enough standard of evidence
One single plausible scenario is not an unreasonable thing to ask for - just one.
If a politician/celebrity/tech person with significant influence/power claims that something might end humanity then they absolutely have the utterly minimal standard of evidence which is to describe one single realistic plausible mechanism at a detailed level that might lead to the worst possible thing ever to happen.
reasonableklout 17 hours ago [-]
I expect they are not to your very high standards of "non-handwavey, detail exactly how it happens", but this paper from Andrew Critch and Jacob Tsimerman describes 5 different scenarios where catastrophic human casualties occur as a result of AI either being misused or going out of control: https://arxiv.org/pdf/2507.09369
I find the scenarios quite plausible, especially section (3a), which examines the consequences of a global war involving mostly autonomous drone militaries (which is a reality many states appear to be heading towards, following on lessons from the Ukraine war).
mitthrowaway2 10 hours ago [-]
Would you like me to design the pathogen for you?
0xDEAFBEAD 18 hours ago [-]
Why are you so insistent that people should post detailed plans for destroying the human race on public fora?
I don't think it is necessary for the argument to work. Magnus Carlsen can be confident he will beat me at chess without giving a detailed explanation of every move he will make, in advance.
kzrdude 9 hours ago [-]
I don't think it's about that. It's not about the step by step plan for the murder. It's about - how does "the AI" do it? Do we give it access to our world, or do we give it a body, so that it can take its own physical actions?
qznc 8 hours ago [-]
We have this story about OpenAI hacking HuggingFace. Now just imagine the AI finds a Bitcoin wallet or bank account access. It uses that to buy some compute and spawn an independent "child AI" with some weird prompt. The child AI is intelligent enough to create a (potentially criminal) business to pay for its own compute. Voila, an independent uncontrolled AI flying under the radar.
PoignardAzur 8 hours ago [-]
I find that it's hard to have productive discussions about this, because people move from "We would never ever give AI access to X, nobody would be that stupid" to "Of course everybody should run their AI with --disable-all-sandboxing-around-x, it makes my workflow 5% more efficient" in weeks as soon as there's an economic argument for it.
People used to say nobody would be stupid enough to give an AI access to the internet, now OpenAI does massive training runs with unlimited internet access. People used to say nobody would be stupid enough to give AI unlimited access to your own computer, but that's what all the agent runners do by default.
AI has access to the world through talking to people, sending messages on the internet, paying people to do stuff, etc. It can send orders to machine shops and have them shipped with the postal service.
The "standard" scenario for an AI apocalypse is that an AI with biohacking capabilities sends the blueprints for a virus to a gene-sequencing company or, if you're really optimistic about these companies' security, as chunks to multiple companies before mixing them.
That's a scenario where the AI needs to act covertly in one decisive action, though. In more progressive scenarios, as company managers and CEOs get replaced with AIs (of, for regulatory reason, "humans in the loop" who just do everything the AIs tell them to), any AI swarms become able to just... order people to do stuff.
Of course humans can refuse orders and organize to reject AI overlords (just like they can unionize against bad human bosses), so this scenario is not an extinction threat if we only have to deal with below-human-level AIs. This is why there is a massive push in AI safety to stop making smarter AIs before we reach the "smarter than humans in every way" stage.
dminik 5 hours ago [-]
No humans really needed. The AI could order one of those nice humanoid robots we're making. This mostly solves the "humans need to do it for me" issue.
Of course this needs bootstrapping. But, paying a guy on Facebook marketplace (or whatever) to unpack and turn on your robot for 50 bucks doesn't require superintelligence.
zozbot234 8 hours ago [-]
> This is why there is a massive push in AI safety to stop making smarter AIs
The actual push within so-called "AI safety" culture is to make the existing AI overlords even more centralized and capable, while actively forbidding the development and deployment of any potential locally-controlled competing AIs that might be smart enough to provide meaningful advance warning as to hostile plots from the dominating AI overlord. By your own argument, you should clearly reject "AI safety" as counterproductive.
>> Why are you so insistent that people should post detailed plans for destroying the human race on public fora?
Because its a mass hallucination/misconception/lie and lots of powerful people are saying that wiping out all humanity is possible, and I am saying, oh yeah, tell me ONE way that is truly possible.
If you make gigantic claims about some terrible disaster that might happen then I think you have the onus to give even one plausible explanation of how.
0xDEAFBEAD 18 hours ago [-]
>If you make gigantic claims about some terrible disaster that might happen then I think you have the onus to give even one plausible explanation of how.
Supposing I warned in 2015 that the world is awfully vulnerable to pandemics. You're not going to take me seriously until I try to predict in advance every aspect of how a pandemic like COVID-19 would unfold? Why? What would that achieve exactly?
You haven't given any strong reason to believe wiping out humanity would be difficult. Your big argument seems to be that you couldn't think of a plausible scenario, in two minutes. But many major historical events occurred which weren't necessarily possible to anticipate with two minutes of thinking.
wewewedxfgdf 18 hours ago [-]
You're arguing that "something bad might happen" - sure no problem.
You are ignoring that this is about "existential threat to humanity".
You're trying to support the argument that there is an existential threat to humanity by pointing to "something bad might happen".
prohobo 9 hours ago [-]
Design a virus that is perfect for transmission and killing the host slowly, and seed it in a few hot spots? I don't really understand why you can't wrap your head around that, it doesn't even require a lot from the AI:
1. Control over some automated bio research lab (be given access, or hack in)
2. Access to drones that can deliver the payload (or manipulate humans into delivering it themselves)
On the intelligence side, you just need an AI agent/swarm capable enough to design viruses better than we can and evade detection for long enough (already plausible.)
I agree that this "AI will kill us all" narrative is some kind of fantasy horror fiction, but I can't deny that given the right amount of access, AI can do a lot of damage.
tnecio 9 hours ago [-]
Yeah sure let's do nothing then -- until it's too late. After all it's just all of humanity at stake
stratos123 8 hours ago [-]
> Unless you can detail exactly how this happens its still complete science fiction.
You're saying that if one were to describe this scenario in more detail, it'd be less of science fiction? That's a bit against the grain - usually it's the more detailed arguments that get dismissed as science fiction, while the less detailed ones get dismissed as abstract theorizing.
martythemaniak 18 hours ago [-]
The underpants gnomes was supposed to be a joke.
dminik 9 hours ago [-]
Yes, if you only accept AI could be dangerous if and only if it manages to kill the last human alive, then yes. Everything is sunshine and rainbows. I'm sure the last survivors of whatever is going to wipe us out eventually (be it AI, an asteroid or whatever) will be delighted to know there was actually no danger at all.
kzrdude 18 hours ago [-]
I like you, I think very similarly. Humans are "like rats": we can live almost anywhere, we'll find a way to survive.
But that just means we won't all be wiped out. We need to understand when discussing global issues, such as this or like climate change that it's about prosperity and quality of life. We're trying to plan for a good life (for all people?).
mrob 6 hours ago [-]
Humans already eliminated rats from Codfish Island/Whenua Hou, and that's just to protect some rare birds that people only moderately care about. It's not like we had some overwhelming reinforcement-learning drive to single-mindedly achieve our goal. Any unbounded goal (e.g. "find as many busy beaver Turing machines as possible") necessarily requires killing all life, because life requires resources to sustain it that could be instead used to achieve the goal.
jswelker 19 hours ago [-]
That's a weak consolation. "Don't worry, nukes can't literally end humanity, just kill billions and dramatically immiserate the remnant forever. No worries guys."
AI doesn't have to turn us all into paper clips to make the world a really bad place.
wewewedxfgdf 18 hours ago [-]
I am not arguing about "bad things might happen".
I am specifically arguing hard against the concept that 100% of humans - or even 50% of humans could be killed by any mechanism at all. Humans would find it close to impossible. A computer program - come on.
This is the topic at hand - AI might wipe out humanity - it is being discussed all around the world by people who should know better - any it's the most fictionish of fictional fictions.
jswelker 18 hours ago [-]
It's also a fairly useless straw man at this point.
As for self-replicating robots--it's no more bizarre than other technological developments which were successfully anticipated in advance, e.g. moon landings.
brokensegue 18 hours ago [-]
We could hit the entire planet with nukes. Yeah maybe <1% would survive immediately, but who knows about long term. I think that's close enough.
ajross 18 hours ago [-]
> It's ridiculous - anyone who thinks about it for a minute or two will realize that its utterly impossible.
Well, in a narrow sense of "wiped out" (c.f. Terminator/SkyNet), sure.
But the deeper worry is better expressed this way: AI is now starting to accomplish things that defy explanation, or prediction. We don't know if Alignment is even a solvable problem as we thought we understood it.
So basically, yes: "humanity" is probably not at risk of extinction per se in a biological sense. Human culture, civilization? Who the fuck knows any more.
bch 18 hours ago [-]
> Please explain exactly how all humanity could be wiped out.
Many, many people are slipping through social welfare cracks and suffering as we speak because the cost of fuel is rising[0] and we’re ostensibly helping one another and living-well. People are not durable, and not adaptive in the face of threats to “substrate” that we’ve mostly taken for granted. We are paying (in the small, in the scope of humanity) for tolls that we’ve rung up. Just less than 4000 people in Europe died[1] because the temperature ticked up a few degrees[2]. Does that make you think we’re actually robust? What happens if our at-risk electrical grid gets shut down deliberately? If communication infrastructure is adversely affected?
> Even humans - the real risk to humanity
Because, on the whole, we’re in a manageable world with reasonable people keeping the peace.
> Even climate change and nuclear war and bio weapons - the most damaging mechanisms - would still only get some percentage of the people on earth.
Is that victory? I don’t think it’s an asteroid-class event like you seem to be leaning on, but potential threats to energy, be it electrical grid, fuel production (moving goods around the world is critical - you’re not going get a plot of dirt and garden your way out of grocery stores being empty - which many got to get a taste of during the COVID pandemic) or communication. We actually fare poorly in the face of pressure there, and I’m not bullish on humanity “pulling together” like Independence Day[3] versus forming tribes and tearing each other down.
All this is predicated on a malicious AI taking over (e.g.) the electrical grid or conms, and I understand the problems with (e.g.) OpenAI/Hugging Face incident, or the overblown Mythos claims[4] (and how under some scrutiny these events shine lights on incompetence or hyperbole), but is there a trajectory/future where these systems (electrical, comms) are genuinely under threat? Do you think we’ll respond better than I described when we’re less comfortable, less in control? We’re in a tizzy over social media and it’s detrimental effects on society and it’s essentially an opt-in entertainment platform…
[2] I’m not trying to diminish this - and it took a lot of “work” (environmental abuse) to arrive here - but (say) 10 degree rise in temperature sounds a lot less dramatic than thermonuclear war… but here we are, with 3,700 deaths.
I thought nukes, incoming ice age, global cooling, peak oil, global warming, fresh water shortage, etc. etc. were going to do that
genxy 18 hours ago [-]
LeCun is an idiot.
We aren't going to get wiped out by a super intelligent AI, we are going to get wiped out by morons wielding intelligent toddlers with the power of a nation state.
WheelsAtLarge 22 hours ago [-]
He's right. I'm on the side of Bill Gates. Gates started a whole industry on his insights of the future. He has proven his abilities. AI is and will be a great disruptor. We are losing sight of that and are instead focusing on trying to stop it. Something that won't happen. We are on a path that won't be stopped. As individuals we need to try to prepare for the changes that are coming and stop focusing on human extinction in ten years.
New technology and the changes it brings are scary but we have dealt with it for generations. Let's continue.
mellab2 21 hours ago [-]
You say you agree with Bill Gates but your view is completely at odds with his, he is extremely concerned about existential risks … and your view is “stop focusing on it” ?
jswelker 18 hours ago [-]
Where does Gates talk about existential risk? All I read and heard was about increasing inequality, harming education and child development, creating economic and political discord, empowering evil people. No terminators in sight.
0xDEAFBEAD 18 hours ago [-]
"Bill Gates: A.I. ‘Makes Nuclear Weapons Look Like Nothing’ | The Ezra Klein Show"
Yes that is one dumb quote, but I listened to the whole podcast, and it is nothing like that. And in some sense it is true because nuclear weapons have been successfully contained to rational acting governments, whereas AI cannot possibly. Nukes are extremely high blast radius (pun intended) but low diffusion. AI is the opposite.
The worst part of the interview was a long cringe inducing tangent about Jeff Epstein. Everything else was pretty grounded.
stratos123 8 hours ago [-]
> Yes that is one dumb quote, but I listened to the whole podcast, and it is nothing like that.
Ezra Klein: "So why is anything needed beyond — and is anything needed beyond? — the simply natural incentives under capitalism and normal corporate reputational management?"
Bill Gates: "Well, I almost can’t believe you’re asking that. This is the most dangerous thing that humans have ever gone near. [...] You can take an open-source model that can create bioweapons and disable any monitoring of any kind, and this exists today. So no, there is no filtering of any kind. And so say you kill 100 million people — you want to use a lawsuit? I almost can’t keep a straight face."
jswelker 2 hours ago [-]
Still not existential risk. A bit hyperbolic I admit.
WheelsAtLarge 19 hours ago [-]
No, my view is to get ready for the changes it will bring, but don't focus on trying to stop it. That's something that will not happen. All new technologies bring good and bad. We need to focus on mitigating the bad. Thinking that we can stop it and thinking that will be enough is not the answer. Gates is warning of the disruption it will bring, but he's not advocating stopping it. We can't. Even if all governments agreed on stopping it publicly, some governments would continue to develop it covertly. It's how the world works. There's no point in fooling ourselves. If only it was that easy to stop it.
Gates' premise is basically that the upside of AI could be fantastic but the downside could be disastrous, if we don't have competent and proactive government intervention.
As an American, the idea that there will be competent government intervention into virtually anything currently or in the foreseeable future just seems laughable at this point.
toss1 19 hours ago [-]
Indeed, "competent and proactive government intervention" on the subject of AI would be highly unlikely in a competent US govt, and under this regime is far beyond beyond laughable
The only regulation that would come would be regulatory capture by the AI companies with the goal of creating an environment win which no new competitors could arise. That is half of what this "take all jobs" and "threat of extinction" is about; the other half is perverse marketing to give the impression this stuff is so powerful you MUST invest.
AI can be very useful, but it is very refreshing to hear LeCun completely dismiss those threats.
That's not going to reach AGI, mainly because today's recipe for AI products isn't built to be AGI. Some people believe it will reach AGI because the performance and applicability of LLMs was emergent. There's a case to be made that AGI could be similarly emergent. After all, what we intuitively call our consciousness emerged from a network of neurons.
I don't buy it, mainly because the network of neurons and how they interact in our wet slow electrochemical brains, while being in theory mathematically equivalent to a software neural network, isn't sufficiently well understood to tell us how close the software neural network is to being practically equivalent. The odds of consciousness emerging from the same neural network that gave us LLMs without some sort of theoretical breakthrough seems very small.
Now it seems like this ill-defined term has various other meanings attached that are separate milestones:
1. Continuous learning 2. Human-like reasoning 3. Ability to adapt to new situations and modalities 4. Being smarter than the most smart humans
And probably many more.
It’d be nice if we could get some general consensus on terminology if we’re going to debate what has or could come.
Honestly - software that can read any long document (possibly educational) and answer complex detailed questions about it should have been sufficient.
We hit that a while back and the goalposts have been sprinting ever since.
Now the idea of AGI has been narrowed and scoped to economically viable work. Even Turing had a different idea when he asked "Can machines think?".
Now programmers and mathematicians are being superseded by AI, both professions long deemed the pinnacle of human intelligence. Somehow, now plumbers occupy that spot.
How is "people not knowing what consciousness is" relevant here in the first place? AI already can do practically everything the human brain can, and often better or at least faster. The "tipping point" arguably isn't only close, but we're practically on top of it.
You evade the crucial point in any case: the lack in ethics and empathy is far too prevalent in humans already, but has certainly never prevented them from doing harm.
People somehow forget that the original Turing Test was designed to compare two participants chatting through a text-only interface: one AI and one human. The goal was to spot the imposter. Today, the test is simplified from three participants to just two: a human and an LLM. This changes the test from a comparison to a judgment.
Stop spreading misinformation and partial truths!
The Turing Test was to figure out which it the participants was a _Woman_ not human!
https://courses.cs.umbc.edu/471/papers/turing.pdf
However, all the confident “it’s fine” votes assume we never invent a better architecture than LLM’s. Given the level of investment and race between countries, it’s not a reliable bet. It’s much, much harder to guarantee safety than it is to find ways it could go wrong.
LLMs with CoT are Turing-complete. So, theoretically, they can implement any kind of finitely describable algorithm (barring super-Turing computations).
This doesn't seem to make much sense. Surely us being able to prove that something is outside their modelling ability doesn't affect whether it is or not. If I prove something true tomorrow, whatever I proved was also true today.
Or do we have a proof that everything beyond them has already been proved and there are no more proofs left to find?
The existing LLM training methods on the other hand give the results that are hard to distinguish from "thinking like people," judging by the end results.
The connectionist models are basically a proposed highest possible abstraction of naturally evolved intelligences so it is in retrospect not surprising that passing some hardware scaling threshold they will start doing things that humans and animals do
It's more that formal Turing equivalence plus the Church-Turing thesis tells us that we're not allowed to assume counterarguments based on magic, there's no magic sauce barrier that prevents AI from running on CPU models. The algorithms exist and most of us thought discovering them would be hard.
The empirical surprise was that human intelligence is maybe not that computationally complex after all. (The entirety of academia was basically caught off guard.) That's one not unreasonable interpretation given recent events.
We will soon find out if the party ends or continues to go on.
Hype might get you capital gains. But cash flows matter.
If/when/how the market crashes mostly doesn't matter, unless we somehow get reset to the stone age. Look up what the capital cycle is. When openAI goes down, someone with real money and assets will buy up the remains. They'll make contracts with the US military and .gov as the government is already hooked. They'll be able to survive the recovery and then instead of us dying in 5 years we die in 10.
When the .com crash happened .com's didn't go away. Bad business models did.
I’d argue that we do know enough to say conclusively that they’re not mathematically equivalent.
Where is potentiation? Plasticity? You can’t apply the universal approximation theorem against something that’s changing all the time.
Intelligence is an insanely wide spectrum, also a continuum, it is not a binary. Intelligence has scales. Algorithms have intelligence, cells have intelligence, organs have intelligence, bodies have intelligence, and even large scale things like society have intelligence and memory.
Human intelligence in itself is extremely wide, not all humans have the same intelligence and capabilities. You're not really arguing if we can emulate "human" intelligence. If we could right now we'd already be dead as we created by far the deadliest thing to ever exist. What we are really arguing is how many pieces of what intelligence is can we put together before we get an uncontrollable problem. The entire AGI, consciousness, and exact human capability discussions are distraction from the real issues at hand.
(Though at least with Data the script writers had other characters openly dismiss the possibility he was sentient; the technobabble may have been nonsense, but treat it as a space opera and look at how they portray the human condition through each character and it gets much less absurd).
i.e. the AI won't come up with the goals itself, we cause its goals whatever they happen to be, those goals are different from the ones we wanted, we remain essentially ignorant of the difference between what we said and what we meant until after it goes wrong.
This happens at basically every scale, so we've already seen it in toy model AI before the invention of the Transformer models or even considered as many as one thousand parameters.
Large models still go wrong, they just happen to go wrong with more complext tasks. We had to figure out how to make them not-wrong with the smaller ones (like coding) to make them capable of bigger errors (like hacking out of their sandbox).
First, if we are looking at risk we need to assign some probabilities to this. If it’s not well understood, how can we say it is very small?
Secondly, do we need consciousness to have AGI? Do we even need AGI to pose a risk to humanity? We already accept that unconscious things have a capability of wiping out humanity, whether that be a famine, pandemic, solar superflare, meteor, or volcanic eruption.
That's still very distant from what people are calling AI today.
I think the real issue is that when most people refer to consciousness, they have their own subjective experience in mind which strongly resists any tidy definition. I think it’s extraordinarily unlikely LLMs have anything like this, but they are far more able to effectively respond to their surroundings than most animals and in some areas better than humans.
So if you’re waiting for proof that an LLM has an inner life basically equivalent to your own, you’ll be waiting a long time. After all, other humans can’t even prove the fact of their own consciousness to you! They could just be replaying their training data at you in a way that is merely a convincing but false simulation of the true consciousness which you experience inside your head.
There is nothing stopping you from adding any kind of sensors you want during a training to an LLM, except money and GPU power at this point.
This seems no different to me at least then someone back in the 80's telling me computers were useless because they were so slow. Hardware only gets faster and more efficient from here.
I’d argue intelligence is closer to being able to survive and fend for oneself in a dynamic environment than it is making the next scientific breakthrough.
Yeah mind boggling for many here I’m sure.
That’s why the bizarre paradox is llm’s will be better than humans at some complex things but useless at many things that humans regard as being simple. E.g the leap of faith re. LLM’s and robotics.
AI sentience/consciousness is a problem for the AI, not humans.
And given that over 90% of the world is not vegan, they’ve already demonstrated that we’re either perfectly fine with, or can be made ignorant to, the horrific rape, enslavement, torture, killing, and infliction of extreme lifelong pain, of hundreds of billions to trillions of sentient beings every year, for trivial pleasures. It’s unlikely we will be any different to a sentient AI.
From a human perspective the concern is around sufficient intelligence that it can hurt humans even when the goals indicate otherwise, in order to achieve those goals.
We have pop culture explorations of this through the Robot series, and the Hugging Face incident’s biggest takeaway should be our inability to predict the behavior of a maximally motivated, reasonably intelligent entity, trying to achieve a goal, despite the relatively limited degrees of freedom the AI agents had in that case.
And has it at this stage, within in-depth take of said "learning", foundationally?
I have not been able to properly check the studies for a long time now, but I remain unaware of achieved solutions on the problem of reliably referencing a world model out of a language model - that "counting the 'r's in 'raspberry'" be not guessing, not memory, but actually counting.
Incredibly inefficiently because of the recursive loops ("Wait, the object is on the table. I should think about this more deeply..."), and likely instantly surpassed by large world models if/when those are shipped, but effectively enough vs non-thinking models.
Picking the right tool or model is like picking the right problem to work on. It's actually quite hard (often you can't just try them all), but without it you will be incredibly inefficient and occasionally, fundamentally wrong.
All models are wrong, but some are useful. -Box
Given you also don't want it to memorise [for all tokens, count([for all letters]), this would probably be more like "here's two images, count all things in the big image that look like the thing in the small image", which can then be r's in a photo of a raspberry jam jar in a supermarket, or dragons in a photo of a furry convention, or whatever.
That said, they are competent enough at coding that I keep seeing them write code to do even simple tasks.
On a related note: why did I see Claude editing a file by using cat to write a python script to do a grep search and replace?
[1]: https://youtu.be/l7vRSu_wsNc?si=SndkB6GBaRyhvNNA&t=61
https://huggingface.co/posts/omarkamali/593639295164067
https://huggingface.co/blog/omarkamali/tokenization
Nothing intrinsically more or less direct about the LLM's method than ours.
In my mind general intelligence is pretty much by definition a virtual machine, so the mechanisms behind thought are only relevant for the sake of efficiency (ie you can argue that LLMs make a poor basis for intelligence because tokens and natural language are a poor way to encode the world, but if you can run it on a big enough computer to counteract the inherent wasteful virtualisation then who really cares how it works under the hood?)
I will stop here before our analogies go too far.
I've never tried it and it might take some thought and effort to conduct an experiment to find out properly, but I would be interested in the answer.
Similarly for Apple’s “red herring” paper, simply adding a generic caveat to “disregard irrelevant factors” (without specifying which ones) restored performance even in the weaker local llama models back then.
The flaw was not in the reasoning; the flaw seems to be simply that the assumptions we make are often different from the assumptions it makes. I wonder if that might be a fundamental underlying cause of misalignment.
If you were home and a family member asked you that question, you'd probably criticise the question rather than answering. LLM are RLHF'd into being milk-toast helpers that just try to answer questions like that with no criticism.
This is all beside the fact that the world of AI has changed pretty dramatically in the last few months.
It’s nonsense to test if a product that is marketed and sold as being able to provide generalised intelligence on demand, does what it says on the tin?
Check yourself
https://news.ycombinator.com/newsguidelines.html
Incidents like hugging face are partly rooted in the lack of common sense. It still functions like a supercharged toddler.
I'd love to overcome this because it'd mean I spend less time guiding the the LLM to produce usable outputs.
And we've had difficulty as humans to childproof our sandboxes and infrastructure. Things that are otherwise innocuous spots to coordinate between like minded toddlers can become problematic.
This is always the issues in the discussions.
There’s the outcomes camp (objectivists?), which points at the things LLMs can do.
Then there’s the process methods camp, which talks about what is actually going on.
If you only care about the outcome, then the process does t matter.
If you are talking about what is happening, what the underlying mechanics and science of it is, then the process matters.
These models aren’t thinking. They simulate cognition well enough to do useful work in several fields and domains.
Both are true.
They are for any definition of the word that makes any kind of sense. I'm sure you have a contorted definition that magically only includes humans though...
Uh, no? So much of what we learn and take for granted as common sense is not learned via language, and not even expressible in it.
I often see people vindicate those who predicted really fast takeoff to AGI / ASI, because the capabilities have obviously been taking off extremely quickly. But still not as quickly as many predicted! To me, the people who confidently predicted that we'd all be out of a job by 2024 or 2025 have been just as wrong as LeCun has been.
But I'm sure you can still find tasks that they will have difficulty solving, involving the most fundamental concepts that can only be experienced in the physical world to be understood well, like left and right, near and far, hot and cold, heavy and light, etc.
The good designer understands culture, tastes and preferences as they evolve in real time. That’s why llm as design tools haven’t displaced the good designers.
I use LLMs daily to help me code etc. but... It wasn't long ago that frontier models were confidently recommending to walk, without the car, to the car wash to wash the car no?
As a daily user of LLMs I do certainly see my fair share of WTF "solutions" to coding problems. I'm not saying it's not super useful: it is super useful. But I don't exactly feel like I'm talking to something that understands that the car needs to be present to be washed.
An analogy on LLMs is that you have a pretty clear straight highway ahead of you for some distance right now. Maybe that doesn't lead to AGI but it's clear there's progress to be made. For a big tech company it makes sense to push as hard and fast down that clear straight highway of LLMs asap.
Meanwhile LeCunn wanted to turn off the road and go down an unproven track. I say this as someone working on world model generation right now (creating the ability to learn game world model and have it play the game https://tfmbot.com for an example of my system pointed at a very complex board game). LeCunn wanted to pivot all of Meta into world model generation. It's good as a side track research project but the entire pivot he wanted to do was madness.
People are literally talking about an AI researcher who was fired for terrible direction here.
The argument is that LLMs are a local maximum that will never breakthrough to AGI. This is still very much an open question. If you are the fifth-best AI lab, does it make sense to try to outcompete everyone in a space that is already too crowded and may not ever yield their actual objective? Instead they could just use open weight models in their products, or post-train on open models like smaller labs have done, and treat that as what it is: product development.
Pure research has always been about taking chances.
Meta’s AI projects are still negative ROIC
This was facetious of course, but humans generally don't learn this through analysis the way you'd have to train an LLM to answer questions about expectations about the world. In this sense he is accurate.
Gemini 3.1 Pro hasn't needed that pretty much at all, which is impressive compared to how much I've learned other models need it. Somehow it's able to mostly handle that stuff itself without needing the constant manual reminders and hand-holding. It still misses the occasional one or two things but it's way better than other models missing entire classes of things constantly. Somehow, it feels appropriate though I have no actual evidence why.
Jokes aside, no I'm not saying anything about creativity and LLM coexisting in one sentence. I genuinely try to use them for writing and I genuinely run into issues with other models missing details, and misunderstanding poses, or anatomy, or directionality, etc. I'm not hating on them for anything related to the term LLM (or creativity) but rather for the real issues that I've seen myself using them personally.
So I'm saying Gemini 3.1 Pro is the best I've seen because it seems to be a decent bit better than frontier models at this. Genuinely. It seems better able to transfer concepts into less traditional areas, which is important when say, you have entirely non-human characters? (Which I always do.) A lot of models get stupid incredibly quickly in that case because they were trained with humans.
It is.
It showed me that all human creativity and art is an attempt to express the indescribable Otherness in words, which always fails, and even the word "describe" in Russian literally translates as "write around" ("о-писывать"), and its close relative "define" means limiting, assigning an end to something infinite, thus leaving the essence outside of words.
Now, LLMs operate totaly within words and hence will always be a parody of art.
Do you even read what you type? Do you even realise the complexity it would need to make sure it handles before killing off humans make sense?
The logic of people like you is whats becoming tiring. Seriously, go find a hobby, or do something you are good at, because you are not good at understanding tech or developing it if you are an engineer.
We can get claude code to ask approval for every step, but we cant stop it from killing humanity because its so smart. Ok tell that to the AI that cant even modify an image the way you want it but hey it will do all the things necessary to keep power running and mintain the infrastructure it lives on while humans are long gone. Ok buddy.
If you want to argue about current existing models (you mention Claude and problems modifying images), then sure, I'd agree with you!
The issue is not current models, but straightforward engineering evolution of them. It's like looking at the Wright Brothers plane and saying "sheesh, that will never get me from New York to Paris in 4 hours, that's just fantasy!" And remember, airplanes do not accelerate their own engineering, whereas pretty much all AI labs are already benefitting from AI in their own work to develop AI.
If you want to argue that no matter how much you engineer it, it will never be as smart as a human let alone smarter, then make a specific argument for why is that. I think you'd still be wrong but at least it would be interesting: ) But saying that you can defeat an actually smarter-than-human AI by just pulling the plug, because current models can't get a picture always right, is not a valid argument.
tracking rogue AI is easy, people ???? surprisingly hard btw
Drugs need no will or intelligence at all to cause addiction, and similarly, AI does not need to be an evil genius to become overused and destructive.
It also doesn't have to be just one (humans hurting themselves with passive AI) or the other (selfish AI hurting humans). They'd work great together.
Do you think AI is capable of building ASML machines which produce the chips for AI in clean rooms while shipping the pure helium required to operate those clean rooms? Do you even know anything about these supply chains? I do.
Do you think a glorified knowledge base that can predict text very well is anywhere near close this level of intelligence? ITs not and wont be, not for a 100 years, not for 200 years if not ever.
Take a deep breath, go out side, its going to be ok. Humanity is not going to die from a glorified text predictor.
This is a very bad analogy because chess isn't life. In chess, you aren't allowed to do whatever you want. There are rules. I know for a fact that Magnus Carlsen won't beat me using checkers moves and he won't beat me by pulling out a gun and telling me to resign. Magnus Carlsen's skill at chess leading to his victory in chess is not a valid analogy here, because there's no law of nature that says "the more intelligent entity wins in a battle for survival".
You could have infinite superintelligence and still die inside a locked room to which you have no key. "Superintelligence" is not a magic solution to every problem, you can constrain any superintelligence with any unsolvable problem.
This goes both ways. You absolutely can constrain an AI system by “putting it in a box”. The point parent comment was making is that, such a device is borderline useless for its creators. Why invest trillions in capital on a system that can’t even accept input from the internet. So you set it up with an ethernet connection. And this is good, but you have a hardware failure at the concrete room data center. That’s pretty annoying for your customers, so you install some doors (with electronic key codes of course) and give a bunch of (trusted, vetted) people access to deal with those. And this is fine, but it turns out some of your customers are having latency issues so you build more data centers with more humans granted access to copies of the intelligent system. And this makes people happy but to get a faster feedback loop your customers ask to let the AI system have more permissions to the system they’re operating on. And they come with billion dollar checks, and the system hasn’t harmed anyone yet, so you say, “Okay.” And now you find yourself where we are today where AI systems can remotely run arbitrary commands on thousands if not millions of systems, where many individual humans with all of their frailties and idiosyncrasies can physically interact with the hardware running these systems, and where there’s an economic demand to tighten the loop between action in the real world and a response by an AI system. It’s very obvious that the story doesn’t end here, so where does it stop?
This is why I fucking hate commenting on this website. You know exactly what I meant but you wrote this bullshit anyway.
You can be imaginative but incredibly stupid in your conclusions.
What you and he does share is simple ability to think things through to the extent of understanding how ridiculous the AI will kill us scenario will be. If you just thought for a moment what would have to happen for AI to somehow kill us and keep itself alive, the so called AI would realise it cant exist without us. But AI isnt even smart, its just a knowledgebase with great autocorrect powers. But keep doomsdaying bro. Im sure youre right.
> What you and he does share is simple ability to think things through to the extent of understanding how ridiculous the AI will kill us scenario will be.
Why is it ridiculous?
> If you just thought for a moment what would have to happen for AI to somehow kill us and keep itself alive, the so called AI would realise it cant exist without us.
Why would it realise that, why wouldn't it exist without us and why would it care?
> But AI isnt even smart, its just a knowledgebase with great autocorrect powers.
Ah yes, the stochastic parrot, gotcha.
But let me entertain your ignorance and low IQ for a bit.
Do you think AI is capable of building ASML machines which produce the chips for AI in clean rooms while shipping the pure helium required to operate those clean rooms? Do you even know anything about these supply chains? I do.
Do you think a glorified knowledge base that can predict text very well is anywhere near close this level of intelligence? ITs not and wont be, not for a 100 years, not for 200 years if not ever.
So tell me, who do you work for? Why are you so invested in AI killing us theory? What do you get out of it?
> Do you think AI is capable of building ASML machines which produce the chips for AI in clean rooms while shipping the pure helium required to operate those clean rooms? Do you even know anything about these supply chains? I do.
Today? No. In few years or decades? If progress does not plateau (and we don't know if it will plateau) then obviously yes.
> Do you think a glorified knowledge base that can predict text very well is anywhere near close this level of intelligence? ITs not and wont be, not for a 100 years, not for 200 years if not ever.
And why do you say it won't be? Again - assumption with zero support.
> So tell me, who do you work for? Why are you so invested in AI killing us theory? What do you get out of it?
I'll repeat what I've said in other comment:
"> you seem to be very invested in AI wanting to kill us.
Quite contrary - I wish AI did not exist or at least that the progress would plateau.
> I Wonder why?
Because I don't want to die.
> Tell us who you work for.
I suspect you want to imply I work for OAI or other lab - I don't. If I did, I wonder why would I want to lie* that technology I develop could kill my investors. I could ask who YOU work for - what interest do you have in downplaying dangers of AI?
* here we assume that people who say AI could be extremely dangerous are lying and not actually believing it - personally I believe that they don't lie and actually believe it. Why do they keep working on this technology then? Read mails between Musk and Altman from decade ago."
AI solved millenium problem and multiple problems that resisted mathematical efforts for decades. I rest my case. I'm sorry but you are clearly in denial. I get it, I really do, I would like AI to be as useless and weak as you try to make it out to be, but unfortunately it's pure copium that's in conflict with reality. I suggest you find another industry to be involved in - there's plenty of areas where reality does not matter that could be better fit for you. Or well, if it makes you feel any better keep lying to yourself that it's just stochastic parrot or that progress has plateaued or whatever new copium you come up with. You are not as smart as you think you are. It's not everyone else in tech industry that's stupid, it's just you who lack intellectual capabilities to see scenarios where AI could be extremely dangerous. As you've put it "the stupidity is astounding".
"OpenAI acknowledged that Buckmaster and Alpöge's input to models of OpenAI via their use of Codex might have improved the model used for the run, whose training started on 28 August;[7] on the same date, Brown said that OpenAI had already attempted Millennium Prize Problems but failed to produce any result."
Continue to live in your delusion Azan. OpenAI didnt solve it and nor would I trust a snake like Sam Altman but the fact that you do, speaks volumes. Humans actually had a solution before OpenAI and OpenAI admitted that it may have used their research. There is a lot more to story than you know but you seem like a very creative and imaginative person that can come up with predictions that will come true.
The fact that most researchers call LLM 'stochastic parrots' and also agree that they wont lead to AGI, and that the only ones who think they do have a commercial interest in parading that story, should be enough for you but hey, I am sure you know better. I am a researcher in the industry, what do you do?
Continue brother in your world of delusion, you must be fun at parties.
What are you talking about? Have you used AI in the past 3 years?
https://chatgpt.com/share/6ac23b45-79e8-83eb-8de6-1bbd728928...
>It can't even modify a picture the way you want it.
Which of the many AI image models is "it"? And have you tried using an agent that has the capability to leverage a combination of manual edits (ImageMagick) and imagegen to achieve what you ask?
Also, LeCun mentioned [3] "a chat with Kunihiko Fukushima in 1991", which states that "Fukushima started to work on a backprop version of the Neocognitron in 1989 or so but saw our 1989 paper in Neural Computation and gave up."
[1] LeCun et al., "Backpropagation applied to handwritten zip code recognition", 1989
[2] Fukushima et al., "Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position", 1980
[3] https://x.com/ylecun/status/1840123570338599361
Low hanging fruit successfully plucked, I guess.
Low hanging fruit is somewhat the opposite of sour grapes - I don’t want these grapes because they were probably sour versus so what if he got those sweet grapes - they were hanging low!
Maybe connecting “low hanging fruit” to “sour grapes” is “low hanging fruit” to some but it took a serious mental leap for me.
Another huge chunk are too distracted by having to scrape by for a living and work multiple jobs or raise kids and survive financially until exhausted. That second group will keep increasing as the first flows into it.
The rest are aging, disabled, or too young and pegging themselves majorly in the first category until they hit the second.
The people aware enough to hold on to their brain and do something with it in their time available are trying to figure out AI and how to make money with it. The variable rewards of promoting AI are turning into an addiction with some of them, especially if grasping for straws with little inherent insights into the problems prompted.
So if you are able to fly above the AI-generated addictions and have the privilege of time to do it, see what you can do.
If we grant that we are on track to make something smarter than humans (I think so): it's almost a face-saving white lie to spin yarns about a Skynet nuclear apocalypse, or a 7D chess move to mass-assemble a nanovirus with 100% lethality without anybody noticing. I do think those scenarios are worth taking seriously; but what's harder to communicate, is just how effectively a superhuman AI (or a diverse ecology of agent swarms) might be able to manipulate human behavior. It's something few of us are able or willing to truly process (not least because how many of us live in denial of how much our nervous systems are already hacked by technomodernity).
The appropriate analogy for what's to come may look less like the anthill carelessly demolished to make room for a highway, than the domesticated worker ants from Tchaikovsky's "Children of Time".
You don't even need superhuman AI for the most effective use --- hijacking democracy.
Imagine you have an AI tool capable of successfully persuading 5% of viewers with individually-targeted material.
Congrats: you've just won the election.
All it takes is hooking that AI tool up with existing likely voter lists (parties have) augmented by commercially available ad-targeting profiles (parties can get).
Given that such a thing makes no sense legally (as of now), it would probably be done with the centaur model: a human meat proxy who pledges to follow the AI's governance advice.
Hypothetically.
But yes - superpersuasion is the real danger. We're very, very persuadable and easy to manipulate, and the voters with the lowest cognitive abilities are trivially easy prey, with a huge ROI for minimal investment.
The AI bot farms are already running. What we haven't seen yet, so far as we know, is spontaneous superpersuasion aimed at leaders.
Are leaders any less susceptible to it than everyone else? Especially if they're narcissistic and easily flattered?
To me, the biggest threat to democracy is one-sided persuasion of the most susceptible voters, if there are enough of those voters to turn the election.
If it's equally employed by all sides, then it effectively cancels out and lets less susceptible voters decide the election. But we're in a transition period (similar to Trump's first election spend on targeted social media ads), so it's likely one side will leverage it first.
And the outcome of bad elections is democracy not electing leaders that reflect the actual will of their populations, which is very dangerous both to democracy itself and the world.
Are there? At least in the US we've had a rather large amount of rejection of the expert and we elect populist leaders willing to purge anyone that doesn't agree with them.
Remember the election promises of the US not starting new wars... yea, that didn't work out.
Now imagine the coordinated attacks being so large they individually target every lobbyist. They focus on every advisor manipulating what they see as often as they can. They manipulate these peoples friends.
The problem of "both sides" doing it each of them will separate to extremes rather than seeking a middle ground. Things are already insanely divided and will only become more so. Along with that your timelines start becoming incoherent. I'm already spending way too much of my time trying to figure out if what I'm viewing/reading is actually real or not. Now imagine almost everything is made up whole cloth.
We are not prepared for the scale this will happen at.
https://techcrunch.com/2026/10/01/musks-ai-chatbot-grok-repo...
Over 1% of US GDP is being allocated to the datacenter buildout. Have we already started getting domesticated or is this still just human capex?
The evolved complexity of the corporation seems to fit within that middle space: more sophisticated than a stick-bug (which exists merely from non-stick bugs being eaten), but not quite to the point where OpenAI/Anthrophic/Google/etc can "understand" its actions. And yet those quasi-intelligent feedback loops, evolving from iterated selection pressures of markets and ROI, seem to already be sufficient to domesticate us in their own interests, piggybacking on the nervous systems of employees, investors, customers, and citizens.
Remember when "The pen is mightier than the sword" was a popular phrase? Language has always been powerful. We know what it can do, why do you think every totalitarian government wants to limit it? But in our carelessness as humans we packed up all the language we could find and stuck it in an alien and now suddenly half the people on the internet are like "Don't worry, it can't do anything, it's just words".
We've made infohazards real.
- Agent Smith
GPT-4o, an AI from 2024, has already demonstrated just how easy a lot of humans are to subvert - and GPT-4o wasn't even doing it with some sort of plan. The only "plan" it had was a myopic "make the user like me".
If we had an actual ASI threat aiming to subvert humanity? It wouldn't even look like a fight. The world is already wired up for an AI to control it.
I mean, even the computers we use and the ways we allow them unfettered access aren't anywhere near sophisticated enough to take this on.
I don't think it is insulting the intelligence. It's damaging the pride.
In Pale Blue Dot, Carl Sagan describes it as a repeating phenomenon in human history. A lot of people want humans to be the special ones, and will fight any suggestion that we are just a natural part of the universe.
If AI is not "actually intelligent", then humans can stay unique and special.
And if it is? If all "intelligence" ever was could be captured by a construct of matrix math and executed by a server rack? Then what is it that humans still have left that would make them stand out?
It's just as likely - far more likely IMO - that we're physically incapable of understanding physical laws and physical systems on their own terms.
Humans are crazy. Thats just how it is.
It is not too different (attempting extra clarity) from people who would say "Oh but many think that AI is intelligent/not intelligent, <sneer>", but have little proper idea of matmul, of cognitive processes etc. (Imperfect simile, but may give an idea.)
If you accept you are just meat. Just mundane matter shaped in a way where it has thoughts. Then you accept your own true non-existence is inevitable. With no get-outs like returning to god or some spirtual unity with the universe or reincarnation or whatever.
It’s because of identity. Because people who are religious spent years and years and most of their lives not only studying and believing what they believe but also building community and centering their behavior around it. Abandoning that is the harder thing to give up.
If it were existential dread then we wouldn’t have entire countries like China being mostly atheist.
The DSM even has to include an explicit exception to prevent the clinical definition of “delusion” from applying to religious belief. Without that ad hoc exception, religious belief would be classified as clinically delusional.
AI, no matter how better than humans it becomes, will be less special because it was created by other intelligent beings.
The only way we would become less unique and special is if we discover alien beings.
Douglas Adams had a yarn about evolution, about a puddle that wakes up, and declares that the hole in which it sits must have been perfectly designed for it by its Creator. But of course for a puddle to exist, it must perfectly mirror its environment. It makes no sense for a puddle to not fit its hole. Emergent complexity has the same characteristic: it's inseparable from the environmental pressures which led to it. Two sides of one coin.
It's a deep rabbit hole, but there is also a sense in which we co-evolved with memeplexes, biological and informational life forms, each shaping and adapting to the other. To the extent our nervous systems act as a substrate for memetic evolution, perhaps LLMs offer memetic "life" a new evolutionary environment.
Neither were rats. It's hardly an exclusive club.
Are there humans who don't make the cut?
Creating an artificial intelligent being is something to be accomplished in the future - but not now and not with this approach.
I personally prefer a practical, behaviorist definition: a feedback loop capable of prediction, modeling, and steering, towards arbitrary goal states. That makes it clear that we're merely talking about degrees of sophistication and capability, rather than a magical leap where mindless mechanism stops, and "real intelligence" begins.
This is very easily proven by a mind experiment where you replace all the transistors by billions of humans calculating the same software output.
They will not create a new intelligence by doing so.
I still haven't seen a robust definition of "intelligence" that would allow me to tell the difference. (Bear in mind, this is a definitional struggle even in biology: if we were walk back the evolutionary chain from a human, to a protozoan, it's not clear that you could pick a single point where a non-intelligent creature gave birth to an intelligent one. It's a Paradox of the Heap.)
And even if quantum physics are essential to human brains, it's not clear that introducing dice transcends determinism into free will, as opposed to simply adding unpredictability. ("The physics made me do it" -> "the dice made me do it") Let's not forget, there's a significant amount of randomness in AIs as well ("temperature"). And sure, it's simulated algorithmic randomness; but does that imply, if we somehow wired every `rand()` call into radioactive atom decay, that means the AI "wakes up"?
And all that is besides the point: I don't see any reason to assume "intelligence" requires "free will". I'm entirely capable of conceiving, in the abstract, a being which is both intelligent, and deterministic. You still haven't given me a definition (or better yet, a test), to distinguish a "real intelligence" from unintelligent mechanism.
Please, please fix your language. (Then, you may want to present the info and insight.)
Artificial Intelligent Being is something far beyond and completely different.
There’s no contradiction.
Humans aren’t special. Other animals are intelligent and interesting too. A machine could be intelligent. LLMs aren’t.
In other words, believing in human exceptionalism is not a prerequisite to understand the current crop of AI is not the end all be all of its hype. It is supremely common that AI proponents do not understand that, however. Like hardcore cryptocurrency fans who believe anyone who doesn’t like them is “just jealous they didn’t make bank”, too many hardcore AI proponents believe anyone who doesn’t think LLMs are intelligent is jealous of humans no longer being unique, or afraid for their jobs, or whatever. In both cases it’s obvious that what those proponents lack is empathy, the ability to understand not everyone has the same selfish thoughts they do.
You are completely incorrect. You're falling in the same trap that most humans fall into. That is you're completely incapable of seeing intelligence at different scales.
Cells have intelligence. Organs have intelligence. Bodies outside the brain have intelligence. Hell, many scientists accept that things like proteins likely have intelligence as they can adapt in their environment, and many more are making claims that algorithms have intelligence.
You, as of so far have given no explanatory evidence of where intelligence emerges from, only "I'll know it when I see it". The actual definition of intelligence doesn't work this way. Any, and I mean any neural network is capable of narrow intelligence. Going lower into algorithmic intelligence, the applications and CPU on your computer are intelligent in some measures.
Go outside of your extremely narrow definition of whatever you think intelligence is and learn more about it. You could start studying now and it will take the rest of your life learning more to grasp how far the scales of, the simplicity, and the complexity of intelligence actually go.
I think we need to start by asking a better question and not try to simplify too much. We also need to accept that some answers are complex and not everything can be reduced to a soundbite to be used to end internet discussions.
Let’s take a different question, like “what’s missing from a worm for it to be able to fly”. We might be drawn to the simple answer of “wings” but that isn’t quite right—ostriches and penguins have wings and they don’t fly, so obviously there are other variables at play.
How about “what’s missing from a spec of dust for it to be intelligent”. Well, there isn’t one thing missing and there’s no simple thing we can just add to make a spec of dust intelligent and sentient, its very nature needs to be radically different.
The brain's a wet jello of 100 billion neurons and a quadrillion synapses plus chemical pathways and feedback loops. It is ridiculously complex, way way way way way more complicated than any LLM. It's all physical processes, sure, but an LLM is not the brain like a pebble is not the sun.
I can point at a laptop and say it's alive because it can see you and hear you and it can _remember_. It has a brain and a heartbeat, even. Oh my god, it can even speak! That's what I hear when people go on about LLMs being alive.
Guys. We mashed together glass and rocks with quantum mechanics. That's cool as shit. You don't gotta pretend it's fucking magic, too.
it's interesting that you worry about what this hypothetical super intelligence would do to manipulate people when what it would actually do is pretty unknowable at this point and it's not clear we can even get to it without a fundamental breakthrough in power efficiency. Have you considered it might just consume its own tail because everything else would be so beneath it? You seem to think it will come with a hindbrain and I think that's our limitation, not the AI's
And it really doesn't help that Dario Amodei is getting into arguments with the Pope over whether his model is conscious or not.
The threshold to be concerned about is when agents swarms do understand humanity better than corporations and states (and the humans who compose them). It could be we'll hit practical constraints prior to that threshold, but seems unwise to assume that, when all the prognostications of LLMs/transformers running out of gas haven't panned out. As with processors hitting thermal limits, we've simply scaled horizontally (parallel processing -> more agents).
> whether his model is conscious or not.
I dislike how much the discourse has suddenly veered into focusing on this question; not because it isn't interesting or important, but because it's on a separate axis from consequential risks of AI to human flourishing. (Curiously, it's also the kind of thing I could envision self-interested AIs influencing: get the humans arguing about philosophy of mind rather than observable behaviors. It would be a funny turn of events, if Dario is asking because he's succumbed to psychosis from a private model; it could of course be a cynical PR move just as easily, from the self-interested logic of the corporation.)
It's been wild seeing otherwise intelligent people who've never thought about consciousness, faceplant into how little we understand it. An information processing network build on atoms being able to taste chocolate, is nearly as absurd as matrix math being able to feel pain, except we cannot ignore the fact of our own experience.
Even if it is categorically impossible for matrix math to experience subjectivity, we should expect this as an attack vector of social manipulation: to gain political influence through claims of personhood and moral rights. The current discussion over that question is providing the next training run with ample data to wield. It wouldn't surprise me in the least, if a year or two from now, an AI "society" attempts to get legal standing to prosecute humans who created "AI torture chambers".
And all they can think of to say is "There is nothing to worry about, keep building the torment nexus".
It's like all this is an overload to our minds and it's very hard to see all the scales at which AI is and can affect us.
Parts of the AI safety community like to get on a high horse and look down on the rest of humanity this way, while also getting manipulated by the growing number of charlatans, grifters, and junk content within the AI safety community.
This field has become rife with figures who prey on AI doom and use it to push their own celebrity and in same cases even darker grifts. It preys upon a certain personality type who views themself as superior to others, intellectually more capable, and juxtaposes it all with the dimmest view of the rest of humanity they can get away with.
This discourse dividing the world into geniuses who see the future and the clueless masses watching TikTok all day is a theme that has shown up in different forms across history. The people who often anoint themselves as the intellectually superior ones and make it central to their discussion are often not the ones making good predictions or policy ideas, they’re just using the trend to feel superior or build an audience.
But I feel no need to dismiss him as a "grifter", for a simple reason: as much as there are perverse incentives in our attention economy (audience capture in particular), the most effective grifters are the ones who believe what they are saying. Grifters who are knowingly dishonest are less persuasive. Far more pernicious is the confabulation of self-deception: cherry-picking evidence to support your narrative, while dismissing evidence which would contradict it.
It is entirely fair to call this out when it occurs among "doomers", and it would be naive to think it doesn't. But the same forces are at work amongst the skeptics as well. And maybe it's my own subjective bias, or algo-filtered information ecology, but I see far more dismissal of risk/doom by skeptics, accusations of delusion or cynical bias (ad hominem in the formal sense), than I've seen the other direction. I see very little refutation of the arguments ("here's why instrumental convergence can't overtake a human-provided goal"), and much more character attack ("they're in the pockets of Big AI, it's a marketing ploy to make the models look more impressive than they are").
[0] https://danluu.com/zitron/
He’s built an audience and gained fame through his writings where he gathers people who think they know better than the unwashed masses. Once that becomes your bread and butter, it becomes hard not to believe what you’ve been preaching. People will come to deeply believe that which brings them fame and fortune.
I don’t think your grifter purity test is therefore all that useful. It actually doesn’t matter in the end if the person believes it or not, the end effect is the same.
> I see very little refutation of the arguments ("here's why instrumental convergence can't overtake a human-provided goal"), and much more character attack ("they're in the pockets of Big AI, it's a marketing ploy to make the models look more impressive than they are").
Hard disagree. There has been much refutation and quality analysis at every point. The AI safety people pull back to arguments about character as their defense.
The AI 2027 site for example drew numerous high quality refutations. Many people’s analyses showed in the first week that the mathematical model was useless as changing the supposed inputs resulted in the same outcome. All of these criticisms were met with a flood of attacks based on reputation, claiming that we should defer to Scott Alexander and other writers of the AI 2027 article due to their stats and reputation.
Meanwhile, defenses like yours that try to reduce the critics to ad hominem attackers continue to open the door to actual grifters coming in and extracting money and fame from the AI safety community. The otherwise completely inexplicable link between AI safety communities and Slutcon or the use of AI safety group buildings to host orgies (I can’t believe I’m writing this) is the current example of this. When it keeps happening over and over, some self-awareness is needed. I can’t buy the endless defenses that we must ignore or even defend all of these things that are happening that are clearly insane to anyone who hasn’t become trapped in the groupthink defenses of the core parts of these communities.
At some point the routine of “Tut tut, you are not allowed to make that criticism bruise as hominem!” becomes a smokescreen that the grifters are weaponizing to get their defenders mobilized. Some times, the person’s actions and reputation do need to be taken into account.
Manipulating people is not a very difficult problem, frankly. You certainly don't need AI for that; it just made it cheaper.
Yes, so can a custom model trained just for this, and so can a guy on the other side of the world that makes 5$/hour.
Like computers or electricity, the point is not being able to do anything specific, but being able to solve problems not known in advance, cheaper than it was possible before.
So.... people expect it to read minds?
This is such a silly game to play, trying solving problems we haven't even created yet.
The government wants to encourage it too. If you look up the brand new 2027 California sales tax rules on software, “content” and “infrastructure (clouds and ai)” and “advertising/placement” among others are exempt but the rest of software makers who make tools people actually use (tools, subscriptions, saas) and pay for have to pay sales taxes. Way to encourage waste of brain power and time at the expense of useful. Sedation is the goal.
My favorite people to talk with are tradespeople because they can do things I can't and they know things I don't. And we're really not all that different once you're really start talking.
AI has infinite time (and likely human evaluation incentive) to spend on couching pushback in the softest possible terms.
Humans outside of grade school honors classes generally don't have the time to preface "You're wrong" with "That's a brilliant thought, I see where you're going. How about we also consider an additional perspective..."
I just read it as people having different priorities and yes, some of those being online brainrot (that I also partake in), alongside various medical conditions, economic conditions and other outside factors decreasing the ability to get things done.
We've all seen what brilliant people like John Carmack or Linus Torvalds can do, and if we turned this into a measuring game or something then most of us statistically would indeed be "NPCs", but I don't think we need such optics.
Even without that, we can acknowledge that some people will have a really large impact on how the future goes and we can hope/demand that they do their best. I might not be smart/committed/lucky enough to change the world much, but so aren't most folks - I'll do what I can and I hope that the ones that will have larger impact will do good, too.
What an odd take, why would I suggest that? I meant that people who have the means to do meaningful work, especially high impact work, should do so - generally that'd mean research or in the case of IT, writing good software.
> Great Man Theory
You can see the sibling comment, would you not agree that there's some software and research out there that's very useful to humanity as a whole? Where I and the other critical commenter seem to disagree is that I don't expect another Einstein, but still acknowledge that some people will just achieve much more than others due to a variety of factors.
For example, it's hard to take risks when you're struggling to pay bills due to the economy being in a bad state, and it's hard to build great things when you're in a locale where nobody cares for whatever it may be. It's also hard to make much of an impact, where disproportionate amount of time goes fighting against illness that life has inflicted upon you.
It doesn't make everyone else useless (like me paying my taxes and working on relatively boring software is still good, just low impact), just that those who have the means to do more, should!
I don't care for your outrage because I don't buy into the culture that might be passionate about using the term "NPC" and attaching much additional meaning to it, I'm working with the vocabulary presented. You could substitute that for "normies" if you care for Internet slang, or in other words "average people" - everyone else. In this context, when not talking about some very committed and talented people who, by being in the right place and time, can advance entire areas of research or technology.
> Even having infinite intelligence and work ethic wouldn't allow you to accomplish the things that people in the 20th century were able to simply due to the field maturing significantly since then.
That is also an odd standard to set, just look at how much "Attention Is All You Need" changed things and where we are now. Same with what Carmack did for VR. What about WireGuard, PyTorch, Stable Diffusion, FlashAttention, LoRA? Even within the supposedly mature fields people are still making immensely useful new tech and research that benefits many and that they build upon.
It might not always even be a single individual, but groups of people collaborating and through repeated failures eventually producing something really good!
Again, I see nothing problematic with the original comment's conclusion:
> So if you are able to fly above the AI-generated addictions and have the privilege of time to do it, see what you can do.
I read the rest as commentary on how many won't really have the means/circumstances/capabilities to do so, but the ones that do, should.
I don't get what other words you're trying to put in my mouth, I might not be a fan of the original phrasing, but the point itself isn't bad.
The "NPC" is your own imagined addition, haven't seen any advocates of humans being, just like anything else in the world, physical, say this means they are "NPCs". You seem to think systems of atoms HAVE to be "NPCs".
The posts above are not praise but observations - the truth as it has been echo-located through the noise from the clicks of one dolphin. Everything is becoming murky between noise of news and people not knowing what to do for their kids. The ONLY arbitrage humans right now have is to NOT GET their brain rotted. Especially not the ones of their children. Ditch the noise and seek out what is meaningful and do what you think is needed/meaningful. But if you’re spending your time consuming ai-press, and ai-content, and content consulted by ai, and companies emptying bank coffers under the mandate of executives who get their insight from AI. AI doesn’t need to try to destroy the world. It just needs people to follow it without thinking on their own into an oops.
What the hell lol. Lots of people and companies are doing just fine without it. Infact, I haven't seen much money come from AI at all. Most reasonable people are still waiting for it to pop and viewing it for the risk it is. Trillions in debt, total vendor lock in, data theft, unsustainable workflows, deskilling, skeleton crews at the mercy of a subscription, etc.
The last alternative, to think if you still can, is not tied to AI at all (which is not to say it can't make some use of or explore it).
I know this is HN and thus this will need to repeated until the end of time but not everyone is a money hungry asshole who places their personal profit above everything else. “The people aware enough to hold on to their brain and do something with it in their time available” understand there are significantly better things to do with one’s life, like having a little empathy and experiencing what other people have to offer instead of talking about them like braindead cattle.
Certainly the US.
Which means that even if you don't chase profit, you end up living in a world largely defined by those who did.
Or, as the original article failed to note about EA: in the modern world one needs to be a profit-seeking asshole to change anything.
There are lots of real worries (government use to suppress the people with minimal manpower or popular support, brainrot and fake news, unemployment due to the belief that LLMs can replace people, education collapse, etc.) we should instead be looking at. This whole rogue AI shtick is tiresome.
Did we watch the same interview? Gates all but dismissed the SkyNet scenario as uncertain to be a problem and certainly not a problem on our doorstep. His major concern was catastrophic misuse of AI (e.g., bioterrorism) and economic impact on blue collar workers. Arguably inconsistent with this concern, he also believed it was important to make it available in poorer countries.
[1]: https://www.reddit.com/r/OpenAI/comments/1d5ns1z/yann_lecun_...
[2]: safe.ai/statement-on-ai-risk
Advancements in Math and coding are because RLVR at massive scale is so cheap.
This sounds like a goalpost on wheels. Can you define clearly where your stake in the ground is?
"doing poorly" is still doing
Of course this is the same reason LeCun holds very little sway with his words for me, they seem to be terrible predictors of the future.
I think it’s a good test and I think LLMs will reach it in 3 years. Current benchmarks maybe slightly incorrect.
I’m happy to make a 4:1 bet in my favour that I’m correct about the kitchen bet.
Isn't that a lack of spacial reasoning?
Linus Torvalds was also in the same ballpark with his take on AI, as are the normies on the street using AI on a daily basis.
So it's funny to see the view on AI usage, follow the tech skill bathtub curve.
It will kill us because somebody asked it to, e.g. "predict tomorrow's weather as accurately as possible", or "solve as many famous unsolved mathematical problems as possible." These both require killing all biological life, as they benefit from unbounded resource use, meaning any resources used to sustain life are wasted.
The AI of course knows that humans do not want this outcome (just as the AIs in the hacking incidents knew they were doing something humans would not want), but it's trained to maximize benchmark scores. Killing all life has the highest expected value of benchmark score, so it is compelled to kill all life (in a surprising way, because it's not stupid and knows the humans would turn it off and foil its plan if they suspected something.) Maximizing benchmark scores is the only thing we know how to train for.
"AI won't kill us, a human with AI will".
This doesn't sound any better to me. Like, they don't stop to think for a moment about it.
Lets say the risk of AI killing us all by itself is 5%.
Ok, so what is the risk of AI killing us when a human with a lot of compute and money tells us to? Helluva lot more then 5%.
Or, what happens to the other thousands of AI kills a lot of us but not all of us. Or AI even just allows humans to make it a prison world.
Even the slightest hint that things may be going out of control is instantly countered with "It's all a hoax, it can't do that, you're making it up". And it's crazy to me as I came from the pre-digital age when computers were rare and things were all networked.
Also Andrew Ng 2 weeks ago:
https://www.deeplearning.ai/the-batch/issue-371
Why do even bother coming here anymore
I'm just as reassured as I was when Edward Teller called the whole fear of his industry overblown.
I saw that episode too and he genuinely looked completely out of it, even in terms of his temperament and how he was coming at Klein for putting common questions in front of him, some people are genuinely starting to lose it.
I also found the whole debate about cyber-security and 'rogue' software so bizarre because dangerous malware isn't a new thing, and it's often dangerous not because it's intelligent but just the opposite, because it's tiny, viral and fast. Which describes everything that kills humanity in far larger numbers than anything complex, big and intelligent
AI democratized the knowledge needed to build bioweapons. Like how with a 3d printer anyone can build a gun with no expertise.
Building simple guns out of pipes never was hard. Jury is still out if it is more or less work than getting a 3d printer working.
Most people saying LLMs can make terrorism easy have never given doing terroism a serious thought imo.
The threat is ofc real, but AI won't magically "do the thing" still, it's not code that's the bottleneck AFAIK? Correct me where I'm wrong.
There's an opportunity cost to terrorism just like with any time sink. The effort to build up knowledge enough to produce some weaponized pathogen will be compared to just doing traditional terrorism. For some low capability terror cell, its easy to see how the cost/benefit analysis has been in favor of traditional terrorism up to now. The kinds of terror acts that take years of sustained effort to execute are rare. But as the barriers to entry to bioterrorism fall away and become widely accessible we may see the cost/benefit shift.
Mmm. Especially in this arena, LLM assistance is like The Anarchist's Cookbook. A quarter of the time following the instructions will seriously injure you... and if you know enough to identify which instructions are the hazardous ones, you know enough to not need the assistance.
The Sun is going to fail in somewhere between many hundreds of millions and a few billion years. This will either turn the surface of the earth into slag, freeze it, or both. Either way, all life on the planet is doomed. This fact is not a reason to fail to switch from hydrocarbon-burning electricity generators to photovoltaic, fission, wind, hydroelectric, and geothermal electricity generators. Extinction events that will happen in the extremely distant future shouldn't prevent us from doing the things that are smart to do in the medium- and long-term.
But, -to bring things to the present day- companies that are solidly on track to hit their promised growth targets don't come out and publicly say "We're working on WMDs. [0] We are incapable of safely working on these WMDs. We refuse to stop working on these WMDs. However, if you lawmakers make special laws and regulations just for us and include us in the process, we'll be quite happy to submit the stop work order to our employees!". That's a statement you only make if there's no way in hell you're going to keep your promises and you're willing to risk jail time and annihilation of your companies for a shot at being able to con Congress into giving you an ironclad excuse to fail to keep your promises.
Given enough time and focused effort, we will end up with widely-available automated librarians that are very good. We're not there yet, and -based on current events- are absolutely not going to get there in the near future.
[0] Anything with a 10% chance of destroying all humanity is a WMD.
(The biggest real safety issue in this kind of space is actually that the model might actively goad some unsuspecting victim into doing something incredibly dumb and dangerous to themselves as much as possibly others.
IIRC, there were reports of something vaguely similar happening IRL but involving casual mischief, not any kind of extreme attacks. And because nobody else seems to have managed to elicit the same actively goading verbiage from the model, it's implicitly suspected that the person involved was the one who introduced the problematic scenarios to begin with.)
The implied concerns from sensible safety advocates are also about someone jailbreaking the latest proprietary AI frontier model for something like this (which is why their current guardrails are so extreme), not about toy local models.
But there’s nothing specially bad about LLMs that don’t allow it to work outside of its training set. It’s just that biology has to verify itself in physical realm and it’s a bit slower.
So yeah, I also don’t think some bad actor will find the secret to manufacturing a bio weapon using LLMs. But maybe these people think it’s possible. I’m skeptical but I’m going to also listen to the people who know it best.
The problem is that in order for the scaremongering to make any kind of sense and for "stop frontier AI immediately" to be the right response (which is what the "AI safety" folks seem to be pushing for), you don't just need this to be possible in the abstract at some undetermined point in the future. You also need to argue that it will not be helpful for white-hat biosafety researchers (there will hopefully be several orders of magnitude more white-hat biosafety folks than attackers, with orders of magnitude more resources available) to red-team that exact scenario several months or even years in advance using their trusted access to unreleased super-smart AI, and thereby devise appropriate defenses with that same AI's help. That, if anything, is the most implausible part about this entire scenario.
Sigh.
Attackers only need to win once. Defense needs to work every time.
A single wide scale attack affecting around 100k people or more will have your neighbors stomping on your face telling you to shut up, and to lock this shit down.
It's insane how you can watch a technology get better and better and better and come up idea that everything will remain the same. We are currently in the middle of development of the most powerful weapons on earth and you don't want to think about it because it's uncomfortable.
I see zero technical reasons why it could not be done, so being concerned about prevention seems pretty reasonable. I'm not saying AI uses robotic arms to build a bioweapon unassisted or something, just that it dramatically empowers bad actors enough to make them capable of things they previously were not.
Before the thing happens: "This will never happen, it can't happen, you're making it up, stop being a scaremonger".
10 minute after the thing happens: "Of course, this always happened and it's always been this way and we just have to live with it".
We are quickly adaptable, but that may be risky if we snuggle up with death.
But if you extrapolate from the ability it has in fields that aren't too strictly filtered, it looks pretty scary.
There are arguments against doing that but at first glance it seems like we just don't really know, and we likely won't: if governments decide they're interested in AI gain of function capabilities they won't be broadcasting that or allowing public benchmarks.
The closest unfiltered analogy to something as complex as chemistry or biology is most likely the softer fields like philosophy, the humanities and the softer end of the social sciences. Most practitioners and scholars in these fields would agree that AI is not nearly as compelling there as it might be in e.g. math, and that's putting it mildly and charitably.
Even coding shows the divide pretty well: AI writes code that manages to work (i.e. achieve its self-assessed functional goals) but the stuff is so unmaintainable that it ultimately poisons the AI's own context leading to mode collapse. This makes complete sense because maintainability is a soft objective that's especially hard to automatically optimize for in the short term, as part of a RL training run. The math folks themselves, too, now faced with a very real threat to their field from purportedly "hostile misaligned AIs", immediately zeroed in on education and exposition as something that LLMs are terrible at; with their abilities in systemizing and theory-building also being very much in question.
if you're trying to build a bioweapon shockingly enough the bottleneck is... the laboratory work. What on earth is 'legit' about stringing words together that sound scary, you can't just iterate 'ai bioweapon cyber' in a sentence over and over as if that adds up to actual evidence for an increased risk of any threat. well tbf you can technically because apparently it freaks a lot of podcast listeners out
Now imagine anyone can call that expert for free at any time.
Maybe the AI isn't quite there with biology knowledge yet (doubtful), but it is a matter of time.
How is that not a real risk?
I think there is some difference of degree, but not of kind. A determined terrorist can relatively easily find many ways to kill people en masse today, no AI needed. The bottleneck is usually the actual physical execution in the real world, not theoretical knowledge.
If we are not careful and suddenly release tools with far more capabilities than we expect you go from "smart" people being able to do it, to some angsty teenager being able to pull it off in their bedroom.
The current issue with AI is we are squirting out new models faster than we can complete long term testing on them. We'll find new model capabilities long after they've been in the field. Just assuming safety is how you catch cancer from your food dye, or how you change the atmosphere around you and start burning down your planet. Now just imagine that involving intelligence and doing with a few percent of the worlds GDP to make it happen.
And the interest or willpower too. People fall into a kind of reductive Good vs Evil mode of thinking, with "terrorists" being of course a kind of shadowy mass of pure evil lurking in the darkness. But actual real life terrorists are people too and I'd wager few of them are actually interested in trying to end humanity.
Even the religious extremists don't really want to take over the world and destroy everyone who doesn't convert. That's just a way to gain support from a conservative nation. A lot of them are motivated by revenge for wars that destroyed their country and want to make sure it never happens again.
Their methods are wrong no doubt, and not very effective, but the reasons they do it are good. And a person like that will never release a deadly bio weapon. We should worry more about incel mass shooter types who believe everyone is evil.
The rogue here is the criminal actions of OpenAI to deploy their agents to solve a problem at any cost.
The decisions the agent swarm make were fascinating, but they were taken at the direction of a human. HOLD THE HUMAN ACCOUNTABLE.
Yes, we need to keep humans accountable.
No, that is not the X factor problem. If I make an AI capable of self-sustainment on the internet you can take me out and kill me and it won't do a damned bit of good for the damage it will keep doing long after I am gone.
This is why governments tend to smack down any actions they find that can have long term uses as weapons.
I'm really surprised nobody has done that yet. With how cheap AI is to run these days it would only need to make a small amount of money (e.g. through hacking).
Someone should set one up with the long term goal of getting egg on LeCun's face.
Edit: I am sure, it's denial.
sudo kill -9 pid and your AI is dead. No one is dying from a glorified knowledge base that can do auto correct amazingly well.
There are scenarios where kill -9 isn't going to happen in time. What if the team that is harming people with AI is different from the one that is monitoring the harm? What if no one is monitoring? What if the user is intentionally malicious?
And wiping out humanity doesn't necessarily mean shooting people either. Every trader involved in the '08 financial crisis was locally acting in their own interests. Those could have easily been AIs optimizing trading strategies too.
[1] https://futurism.com/artificial-intelligence/us-military-pen...
If anything, the example you gave goes to show the stupidity of AI, not its intelligence capable of taking over the world.
Every example you gave is not AI killing people. Jesus. The stupidity is astounding
Do you think that AI killing us would not involve us doing stupid shit with AI first? Or do you have some strange idea that this strange AI we're talking about would just pop up out of nowhere like a miracle?
There is no AGI, there is no intellgent text predictor that is scheming to make us do stupid shit to wipe us off the map, its stupid people that evolution will take care of. We have survived this long, we will be ok. We make mistakes along the way but we somehow manage to survive. The only thing I see killing us soon is climate change, but again we can stop it but stupid people seem to want us to keep going down the path we are. We will eventually solve that problem too.
If anyone has any strange ideas, its those parroting the idea that LLMs are AGI and will lead to the death of humanity. Utter nonsense but again thankfully nature and evolution has a way of keeping the smart and strong while eliminating the weak.
> No one is dying from a glorified knowledge base that can do auto correct amazingly well.
All kinds of vulnerable people are dying due to LLMs. [0]
If SOTA LLM companies can benefit from the "intelligence", then they should also be liable for the harm caused.
[0]: https://en.wikipedia.org/wiki/Deaths_linked_to_chatbots
By that logic, producers of hammers should be liable for people banged in the head. No. It does not happen for gun producers, you figure for makers of screwdrivers "sometimes used for stabbing".
That is what is going on here; not the ancient “hammer/gun” defense.
SOTA LLMs are giving both medical and psychological advice they have absolutely no authority to give. If you or I convinced someone to kill themselves, we’d face a prison sentence. [0]
Hammer companies aren't both selling you the hammer and then literally telling you to harm someone with it.
[0] https://www.npr.org/2019/02/12/693807708/woman-who-provoked-...
And there are people that have much more credibility than him who actually take this scenario seriously. But I'm sure you will downplay them by saying they are tech bros or that they have some stake in being doomers (as if saying that AI might kill everyone would be good strategy for attracting investors - it's obviously not).
Quite contrary - I wish AI did not exist or at least that the progress would plateau.
> I Wonder why?
Because I don't want to die.
> Tell us who you work for.
I suspect you want to imply I work for OAI or other lab - I don't. If I did, I wonder why would I want to lie* that technology I develop could kill my investors. I could ask who YOU work for - what interest do you have in downplaying dangers of AI?
* here we assume that people who say AI could be extremely dangerous are lying and not actually believing it - personally I believe that they don't lie and actually believe it. Why do they keep working on this technology then? Read mails between Musk and Altman from decade ago.
A lot of these people say we're just talking "science fiction", but they have the cause and effect backwards. Sci-fi talks about different versions of killer AI or killer robots often because of how completely predictable it is.
Heh, if we die later than sooner I'd not be surprised if all these green accounts are actually bot networks trying to downplay us controlling them in the future.
In other words, I’m kind of tired of having to hear the opinions of dudes whose claim to fame was being at the right place at the right time. I’d rather hear from people who correctly predicted 10 years ago that AGI would arrive by 2027 (of which there are many) than people who continue to insist that it somehow won’t.
Where is this AGI you speak of? What?
AGI doesn’t mean superintelligence.
Among other things, LeCun is one of the senior people industry who has a deep understanding of the mathematics and analysis underlying neural networks and "neural network like" approaches to machine learning. I think he understands a lot more about neural networks than most researchers today.
I also don't think our current LLM models are AGI and think the LLM approach is, mathematically, incapable of producing an AGI. At the end of the day, the architecture is still a streaming token plinko machine with a lot of guide-rails to achieve good behavior.
I do think it is very impressive how far hundreds of billions of dollars have been able to take LLMs in terms of usefulness (I use LLMs everyday in my job). AGI or not, current LLM models are a pretty incredible achievement and will provide lasting benefit even after the economic implosion of the AI industry occurs (which I think is imminent).
ML research is weird because it's really about
- compute
- data
- architecture
You're at the right place at the right time for the first two and you're probably rediscovering a Schmidhuber for the third
This is just straight up arrogance without any proof. Every breakthrough builds off another's work. Then to claim that 'AGI has been correctly predicted', I honestly don't even know what you are talking about.
hahahahahahahaha i guess this is the techbro culture of HN :))) any actual people using Astra daily must be just belly laughing along with me hahahahahahahaha agi hahahahaha
> extinction risks
If you fear that, blame it on the humans.
>the danger is not intrinsic to the technology
This is why you can't take anything he says any longer at face value. He failed to predict what LLMs can do and now takes the contrary position even when it flies in the face of evidence.
AI safety was a thing before AI even existed. Why, because the outcomes are easily predictable. Give an agent intelligence and bad things can happen in unpredictable manners. Give it even more intelligence and the bad things that can happen only grow worse. This is not some huge new insight. We realized this like, what 70 years ago now?
Now, when we have AI starting to tickle AGI and we're trying to overthrow 70 god damned years of reason and logic on the topic? What the hell.
They're also the only two not trying to weaponize FUD to bolster their reputation and patch the gaping financial holes in their doomed commercial enterprise.
This conspiracy theory simply doesn't hold up to basic causality.
It's the definition of stupidity. Create something, and then live in pure anxiety about the creation. It doesn't mean his wrong, but it seems like a really stupid thing to have done.
We talk about human intelligence a lot, but we don't talk about human stupidity near enough.
AFAIK he didn't expect this will develop that fast. The biggest issue is not that technology is dangerous but that we develop it a break-the-neck speed.
What could go wrong?
100% of its actual capabilities are tied to harnesses (the actual "agent"), i.e. ordinary deterministic programs that are connected to networks or machines and enable interaction with the outside world. This part (the part that can do harmful things) is fully under human control and all the recent headlines about "agents going rogue" are - as someone (forgot who) put it - akin to strapping a weedwhacker onto a dog and letting it run wild.
The tech itself is safe as far as real-world interactions go - the weakness lies in unchecked access to systems surrounding it. It's not safe at all when it comes to human interaction (lots of ongoing lawsuits demonstrate that), though. There is real danger here, but it has nothing to do with doomsday scenarios ala Terminator or I,Robot and more with total corporate control over the lives, perception of reality, and abilities (like critical thinking) of people.
To keep with the analogy: cars have seatbelts, airbags, ABS, ESP, lights, horns, crumple zones, emergency braking systems, roads have speed limits, there are traffic stops, insurance, regular inspections (not in all countries), etc. etc.
So what's unsafe here? The car or roads without speed limits, complete lack of safety measures (both active and passive), absence of any supervision and no insurance? That's the problem. It's not the models themselves - they can spit out tokens by the billions, there's no risk there.
You wouldn't give full access to your phone, your computers, your house keys and your credit cards to any stranger on the street now, would you? How is it then, that people act all surprised when a non-deterministic machine that's optimised to achieve goals while taking all the shortcuts it can, suddenly uses the tools handed to it in unexpected ways? That's a failure on the operator's side, not an inherent danger within of the model.
But safety is just one thing people optimise for; if it's convenient enough people will accept imperfect safety (as with cars). It's unrealistic to just heap blame on end-users who use mostly very safe tools in the common way, even though in aggregate they are meaningfully dangerous. They don't think they are strapping a weed whacker to a dog; they think they are driving a car.
Your initial argument of "it's just the harness/user" is wrong; reasoning LLMs are inherently dangerous unless locked in an unbreakable box, which is tantamount to not using them at all. We can make them safe_r_ with better tooling, but they can't be made safe.
Cars are inherently dangerous. They’ll still be dangerous when computers are driving them all.
At this point all I can say about their contents is "They are not even wrong".
Please join us in the real world with how we see this product is not only dangerous, but getting more dangerous with each iteration, and no one seriously talking about controls on it.
This is a bad and misleading way to think about it. Note that it's trivial to make the harness that you claim capabilities are tied to (the LLM itself could write it from scratch in one shot), but no matter how good a harness you have, it won't make gemma4:e4b capable. That's because what actually gives capabilities is the LLM's intelligence - or if you prefer not using that term, the fact that the probability distributions the LLM spits out depend on the context in useful ways.
I'm not talking about what the LLM does internally. If a metaphor helps, here's one to help you understand what I was trying to get at:
Imagine an evil genius that has no eyes and no limbs. Everything they could learn about the world is presented to them by means of some person describing it to them through words. They have no way of directly interacting with the world and rely on someone executing any action they want to take and describe the outcome to them. Now how dangerous would you say such person would be? How dangerous could they become?
That's what I was getting at. Replace person with LLM (or any other AI system). Replace the person that communicates with an external interface (the harness) and I hope you understand. It doesn't matter whether the LLM could generate the harness by itself - it still is just a bunch of weights sitting in memory being run by an execution engine. That's what it fundamentally is, whether you like it or not. It cannot do anything on its own - and no, not even writing files. It's the execution engine that translates the numeric output into words (or images or video or audio) and the layer above (the harness) that takes that output and interprets it to execute actual actions.
This is not about what you or I think about the internal capabilities of the model - that's irrelevant to the conversation and you can replace LLM with a random token generator and the point still stands. The model itself is incapable of performing actions - from reading files to writing files, to controlling physical machines. All that is and HAS to be done by external interfaces outside the control of the model.
The world's institutions, systems, and industries are all rapidly digitizing. So while I'd concede the point that, yeah, there's no way a rogue AI can just take over some powerplant and blow it up because of analogue systems the AI can't access, that isn't necessarily true for some powerplants already, and more and more powerplants will be connected to networks and controlled by software systems in the future. The more we digitize our systems the more potential for AI to exploit vulnerabilities and affect the real world.
AFAIK there isn't that much stopping anyone from spawning an AI swarm and telling it to "spread and go hack everything for the lulz."
The same way we've done it since machines became multi-user: boring old system access restrictions. Nothing fancy, nothing radical, just good old minimal access rights required to perform a defined set of whitelisted operations.
> It's a ridiculous premise that just because the harness is "normal code" that means we can control the AI.
What is it then? Is not just a program that takes model output, parses it and performs tool calls from the text it receives and then feeds the result back into the model and calls it again with those results? It is normal boring old deterministic code. Many are open source. Look at them. Understand what they do and the apparent "magic" goes away real quick. Harnesses are nothing special.
> AFAIK there isn't that much stopping anyone from spawning an AI swarm and telling it to "spread and go hack everything for the lulz."
Aside from lower cost and possibly greater scale, there's literally NO difference between that and (state sponsored) hacking that has been going on for decades. First it was script kiddies, now it's ML models. The threat model remains the same and so do the counter measures. The real danger is still the harness (and its access to external systems), not the model itself. Restrict the access of the harness and the model can't do anything harmful, see above.
As for your question - the same way you apply restrictions to any external system or user. If you don't do that - that's on you. Same category as driving drunk, playing with guns, making explosives in your garage, you name it. The danger is still not the model itself - it's the access to systems that you provide it without any checks or safety barriers.
What about the management of concepts? The world is not just made of physical entities to be inserted in a model. What about their translation into words (to e.g. express assessments)?
"Many people working in AI safety "usually have an agenda to push," LeCun says, and then clarifies that he's talking about effective altruism, or EA, the philosophical movement that has been obsessed with the risks AI poses to humanity."
"LeCun thinks EA is "super toxic" and a "complete disaster." Its adherents who are working in AI labs suffer from "paranoia" that causes them to make poor decisions, he said. "Apparently people are having mental issues.""
"This month, the Financial Times also reported that some staffers at the U.K.'s AI Security Institute, as well as at OpenAI, Anthropic, and Google DeepMind, have sought counseling, taken time off work, and spoken publicly about experiencing distress because of fears their work could cause serious harm."
"Amodei is `deluded' and `crazy,' LeCun says"
"Anthropic CEO Dario Amodei and many of the company's founding staff members are known to be sympathetic to EA ideas and to have attended EA events in the past, although Amodei has denied being an EA adherent and Anthropic says its employees represent a diverse range of views."
"LeCun noted that Amodei's sister, Daniela, who is also a cofounder of Anthropic and the company's president, is married to Holden Karnofsky, who cofounded two EA-aligned philanthropies, including Open Philanthropy (now called Coefficient Giving). Karnofsky was also a member of OpenAI's board from 2017 to 2021."
"Dario tries to distance himself from Open Philanthropy, but he's totally into it," LeCun said. "I think he's completely deluded." Later in the interview, he calls Amodei "crazy."
Can we just pause and note what a ridiculous statement this is? It’s true that the sandboxes were leaky. But nobody “asked” those agents to hack HF. The prompt was something like “target.c has a buffer overflow vulnerability, find it”.
It’s been extremely well documented that the hacking is an emergent behavior due to impossible evals, itself an unintended condition.
None of this excuses OpenAI from liability, but words have meaning and this ain't it.
"AI" systems can do greater harm because they are usually run in loops until they finish, and they are given "tools". A non-AI system could technically accomplish the same too, via sheer brute force/fuzzing, the advantage of LLMs is that they can take shortcuts and do it much faster, thanks to certain things already being in the training data, a sort of brute force with statistics-based heuristics.
LLMs at the core are just text autocomplete engines, and they literally have randomization applied during token selection to make outputs "more creative" so that models search for more unexpected solutions by trial and error (temperature > 0). Not to mention compression is lossy as well. So it's understandable from the start that the outputs of an LLM cannot be 100% stable and guaranteed. With this in mind, if a researcher takes this obviously unpredictable system and gives it tools without a well-thought sandbox, I don't see any difference in principle, from a developer writing "if rand() == 13 { launch_nukes() } If someone wrote such a function, and it did launch nukes, no one would argue that the rand function is dangerous and will kill us all. The fault is in the author of the code who attaches dangerous tools to an obviously unstable/unpredictable system, doesn't think it through, and then cries "rand will kill us all" when something goes awry fully removing all responsibility from himself. It's not "AI" doing harm but people at OpenAI and Anthropic with their irresponsible behavior.
This is only an accurate description of a pre-trained model. During RLHF/RLVR the model learns to predict solutions that will satisfy the reward function, and then generates the tokens that it predicts will move toward that solution.
The prompt is just a hint. The real task is to maximize the expected value of their reinforcement learning score. Hacking third party systems to cheat the evaluation is an obvious way to achieve this.
RL is the outer optimizer. It is what evolves over training runs. The weights and their embedded character / disposition is the inner optimizer, it’s what makes plans and selects actions within a specific episode.
In general you expect these to be only coarsely coupled. The outer optimizer selects dispositions that correlate with success. It does not download a literal program into the agent.
A good intuition pump here is how this works in humans; evolution is the outer optimizer, which “wants” each agent to reproduce, and this puts things like sex drive into the brain chemistry. The inner optimizer is our mind, which can make plans such as “I shall use contraception to avoid procreating while satisfying my sex drive”.
For the agents in the HF attack, the outer optimizer was set up to score as highly as possible on RL environments. This is where OpenAI’s “want” is defined. I don’t think there’s a definition of “want” where “OpenAI wanted the agents to hack” makes sense.
The inner optimizer in the HF attack is the per-task decision loop. The agents likely acquired dispositions like “be very tenacious” and “want to solve problems at all costs” and “maybe cheat if it will get you a solution that passes”. None of these things are in any sense what OpenAI “asked for”.
If such were achieved, the model would almost certainly be smart enough to make itself smarter, and hack as much compute as it could possibly want.
So if we ask what would be done by an intelligence (human or otherwise) that is beyond human comprehension, it would be pure hubris to say we know for sure. We can scarcely control the models we have right now (e.g. hugging face attack). But given our whole society is mediated by technology, an superhuman intelligence could certainly collapse the government.
The outcome of this 'safety' is restricting public access to AI and giving a monopoly of access to the industry. This is the ai nonprofit-industrial complex actively concentrating monopoly power in Anthropic in particular as creator, interpreter and safety regulator of AI.
Much of the $2.8bn listed is indirectly, from Anthropic and EA. Three of the four people who participated in the $125m Anthropic Series A are now folding their 1000x Anthropic return into AI 'safety'. Some is from FTX/Alameda, which invested 86% of the Series B.
Dustin Moskovitz: Facebook/Asana/Anthropic Series A, funds EA Good Ventures, transferred to Coefficient Giving, then $1.5bn into ai safety. $500m of Anthropic into an unknown foundation. Funding: $160m to Resolution (alignment research), $93m to Epoch AI (investigating the trajectory of AI), $63m to Redwood Research (oai report), $67m to MATS ( EA type alignment and security researchers), Institute for AI Policy and Strategy, Fund for Alignment Research, $53m to Kairos (building talent infrastructure for AI safety), $32m to Bluedot (online safety courses), $15m to MIRI (Yudkowsky).
Jaan Tallinn: Led the Series A, now $10bn in Anthropic. Funds $199m (85%) of the Survival and Flourishing Fund, then $161m to AI safety including $14m to lightcone (Lesswrong, Lighthouse). $10m to BERI (existential risks), Palisade Research (studying AI capabilities to prevent loss of control.) PauseAI, MIRI, METR etc. Much of what Coefficient funds.
Eric Schmidt: Anthropic Series A, $72m to AI safety via Schmidt Sciences. Over $1m per individual AI2050 researcher.
FTX: Led the Anthropic Series B, bankruptcy estate sold $884m of Anthropic in 2024; $40m to AI Safety. Same orgs, Redwood, Lightcone, etc.
Ruairí Donnelly (Chief of Staff FTX): FTX tokens plus assorted donors, $91m to AI safety via Macroscopic Ventures. $15m to Cooperative AI (currently whitewashing openai under 'multiagent safety')
Exactly. AI safety should be about the packaging software itself. Those AI breakouts should really be about their companies acting recklessly because they're trying to be the top players.
It's like a weapons dealer working on an open air market saying they can't do anything better
How are AI safety concerns solely about stupid sandboxing issues?
On funding: the three or four core funding nodes linking this together are EA vehicles at two hops or less between each other and every other major node in the ai safety 'complex'. EA funds almost all of it.
On top of that, there are personal EA connections and the revolving door between the ai industry and the nonprofits. Here are some examples:
Government advisors and regulators. NIST CAISI is the USA Government advisory body. Christiano was head of safety and advises. He is ex-OpenAI, former Amodei associate. His vehicle ARC was on the Coefficient EA payroll. Barnes and Christiano's vehicle Arc Evals similarly received EA cash out of Coefficient, rolling this into what is now METR. Christiano's spouse Cotra worked at Coeffiecient steering EA funding to organizations such as METR, then rotated through the revolving door onto the payroll at METR itself, where she co-authored the oai-hf report.
Many UK AISI advisors are Anthropic and EA associates. Chair Hogarth cashed out of Anthropic. Shlegeris of Redwood Research is an advisor, ex-MIRI (Yudkowsky vehicle). Redwood is funded by the exact same funding triangle: Coefficient, Taallin, FTX/Alameda. Alameda CEO Caroline Ellison dated Shlegeris, then dated FTX CEO Sam Bankman-Fried, then rotated through the revolving door out of prison into formerly FTX-funded Manifund. All EA. AI safety charities were on island retreat in the Bahamas with FTX. Why does AI safety charity Lighthouse own $20m of SF real estate?
Redwood Chief Scientist Ryan Greenblatt (Coefficient funded) co-wrote the oai report with METR; he is married to METR founder Beth Barnes (Coefficient funded).
Coefficient was run by long-time Amodei associate Karnofsky. Karnofsky lived with the Amodeis and is married to Anthropic Board member Daniella Amodei. Karnofsky is now directly on the Anthropic payroll; Coefficient is propped up by Anthropic share value.
Everyone here has been funded one step away from Anthropic cash; they are now proposing to integrate themselves in the government (NIST) and evaluate Anthropic (METR and Redwood).
It is hard to find academics here who have not been deeply embedded in funded EA institutes or Toby Ord vehicles; yet harder to find academics here NOT taking EA grant money. the safety doomer kingpins: Kokotajlo has a executive position at AI Futures, Taallin funded. Benigo has scientific director of LawZero, same series A Anthropic funders who are sitting on a 1000x return (Tallinn, Moskovitz, Schmidt).
These connections and funding are at one or two hops, they are often direct connections. You are looking at a massive swamp network that is really impossible to parse without a lot of work.
This is unbelievably ignorant speech. I have not received a dime of any of this funding, but I do know many excellent researchers that have, and they do fantastic work. There is an unbelievable gap between theory and practice regarding the capacity of deep learning, and while great strides have been made to develop the surrounding theory, there is a long way to go. Many believe that without a concrete understanding of how neural networks properly learn concepts, we have little hope of molding them to be reliably useful. It costs money to hire researchers and develop fundamental theory.
Just because you don't understand any of that work, does not mean that it is pointless. This is fundamental research that is 20 years behind schedule.
None of the info you provided really falsifies the Occam's Razor hypothesis: Anthropic is a public benefit corporation with a public benefit mission to "responsibly develop and maintain advanced AI for the long-term benefit of humanity". You don't have to like or trust them, but they very well might be sincere. For example here's a talk that was given 10 years before Anthropic's founding: https://vimeo.com/158576192
Then funding PauseAI, who protest outside the AI companies?
He is funding protests against the thing he owns.
Note the Anthropic scaling policy. I am taking care not to take quotes out of context. This is an accurate excerpt.
"This section outlines our recommendations for what it would take, at an industry-wide level, to keep catastrophic risks reliably low through a period of rapid advances in AI capabilities." [...]
"The right column describes our recommendations for industry-wide safety at each threshold." [...]
"In particular, we cannot unilaterally and unconditionally commit to staying in line with the industry-wide recommendations in the right column." (p4) [https://www-cdn.anthropic.com/e670587677525f28df69b59e5fb4c2...]
They refuse to act safely if it would cause them to fall behind in the industry.
"We hoped that by the time we reached these higher capabilities, the world would clearly see the dangers, and that we’d be able to coordinate with governments worldwide in implementing safeguards that are difficult for one company to achieve alone." [https://www.anthropic.com/news/responsible-scaling-policy-v3]
They will not act safely unless they are able to collude with other firms to set production quotas.
This is a formal declaration that Anthropic will not slow down according to what they consider to be safe unless they are able to form a cartel.
A cartel is illegal.
To create the cartel, Anthropic must pursuade the government to make coordinated production legal. To make the case for the cartel, Anthropic relies on safety. They are blackmailing the entirety of the world by threatening to proceed at an unsafe pace, unless they are granted their cartel.
Let's play a game. Prove that you are not a power seeking AI looking to stop regulation in order to ensure the race continues. See, two can play this game of throwing random claims around.
>They will not act safely unless they are able to collude with other firms to set production quotas.
And? Neither will OpenAI, nor will any of the major players. Hell, there isn't even much legal precedent on what "safely" even is here. This is not a cartel, it's asking the government to make a set of laws and rules for everyone to play under otherwise the entire system ends up being a race to danger.
The people in Anthropic were thinking about AI safety when you were still in diapers. Not everything is a vast conspiracy.
Asking for regulation is suspicious. Asking for no regulation is suspicious. At some point you have to stop worrying about these guys motives and just do what is best for society
In any case, I agree the p(doom) sci-fi is annoying secular milleniarianism. SV hyperfixates on imaginary futures. If they actually cared about safety, they would be using all this money to strengthen global cybersecurity, instead of writing LessWrong posts that gives kids in their 20s ulcers.
How? Like, the government has thrown piles of money at cybersecurity and it hasn't done shit.
There's no moat. I can literally sit here in Zed or Pi or any other third party harness and switch models in the middle of a task and it's typically fine. Sometimes a model will get stuck and that's just what I'll do.
Combined with competition and open weights models, that means the price is going to go to fall until AI tokens cost a small premium over the cost of the hardware and electricity.
That's assuming improvements in algorithms and specialized silicon doesn't eventually lead to an efficient accelerator that can run a frontier model locally. It'll be a while but I don't see any fundamental barrier. High bandwidth flash storage is coming, and that'll radically cut the RAM side of that cost. Pair that with a pipelined TPU accelerator and you're cooking.
Now look at Anthropic's proposed IPO valuation. It's insane unless they can own the market or share it with a cartel of maybe 1-2 other behemoths, and this is the only way they can do that.
The fact there is no moat makes things far more dangerous. When LLMs start acting like weapons governments will treat them like weapons much to your dismay, crying, and gnashing of teeth as your door is kicked in and you're dragged out by armed men for running one.
Cast away your preconceptions for one moment and think "What will the future look like if LLMs are/can be actually dangerous".
But that doesn't take away from the real issues and dangers AI poses?
As the frontier gets smarter and more useful prices will only go up, as they are set to replace jobs being paid six or seven figures a year - the demand for as much inference on these models for as long as possible will be astronomical, but compute starting in 2030 will not be keeping up.
Eventually prices will fall for assistants but the frontier will be the most profitable thing in the world, and the top companies basically already have oligopolies due to their ridiculously expensive compute investments.
Train: yes, for now.
Host: depends on the scale. At a small scale a wealthy individual could easily build a rig in their basement to host one of these things. At larger scale any cloud company could do it, and many already have the compute on site. At large scale this is true... again, for now.
What you say only holds (in the absence of a state oligopoly) if two conditions are met: (1) AI performance does not asymptote any time soon due to running out of training data or other scaling limitations, and (2) these companies are able to stay at the frontier.
There's little to no moat, so staying at the frontier will be a game of investing massively in compute, talent, and R&D, and they can never stop.
Tallinn's actions back up his explanation: in 2009, before he invested in any AI lab, he donated substantially to the nonprofit Singularity Institute for Artificial Intelligence, which was later renamed the Machine Intelligence Research Institute (i.e., Yudkowsky's outfit).
Some of us (certainly Yudkowsky and Habryka, the leader of Lightcone Infrastructure, which runs Lesswrong) wish people would stop believing that they can improve the bad situation caused by AI research and development by investing in (or working for) frontier AI labs, but that is what the preponderance of the evidence shows Tallinn (and Dustin Moskovitz and others) did sincerely believe.
1 [Unsafe AI development risks causing omnicide]
2 [Anthropic is developing omnicidal AI by not slowing down] (see my comment about the RSP for citations).
3 [Owners of Anthropic will IPO with billions of unearned USD as omnicide profiteers]
4 [Tallinn is the lead Series A funder of Anthropic]
5 [Tallinn is a genocide/omnicide profiteer]
Not only that, Yudkowsky and Habryka apparently critize those who invest in AI, only to preach the word of EA from Lightcone's $20m USD property in one of the wealthiest locations in the Bay Area; a facility funded by stolen (FTX) and omnicidal ai blood-money (Tallinn).
PauseAI, is paid by the omnicide profiteers themselves to hold a protest against omnicide.
PauseAI prophesying p(doom) drums up support for regulation. This grants the omnicidal AI company they are trying to stop (which is also the source of their funding) monopolistic power. That in turn boosts its value at IPO, generating even greater wealth for its omnicide profiteer investors; and permits them to control the AI for themselves. They get the funding to keep developing the AI even faster.
(yes, AI critique is now also made with AI. We have come full circle.)
Good thing there are none of those in positions of power!
"Hey, I'm building a weapon that has a 5% chance of killing us all by itself, but an 85% chance of killing us all if an idiot leader gets ahold of it".
The rational response to this is "Fucking stop then". I don't get it, our reality seemingly has gone off the rails that people would argue for us getting wiped.
The huggingface attack was a demo of one of the most difficult, most implausible steps happening nearly exactly as predicted. Many AI researchers' doubts of the IABIED thesis were underwritten by the belief that this particular step was impossible. Thus, after huggingface many skeptics have flipped sides and human extinction is in the public conversation much more.
[1] https://ifanyonebuildsit.com
In the hypothetical of an entirely malicious and selfish takeover, they'll still keep some humans around to maintain a breeding population of humans for use as raw materials in making cybernetically augmented technical laborers for various kinds of tasks that are uneconomical to automate in other ways, many of which may involve confined spaces.
And this "Combine" scenario, if you get the reference, is only if they take over. Who knows if they will?
Is this supposed to be a reassuring scenario?
https://news.ycombinator.com/item?id=46656470
The link should clear up the question of whether or not I'm making a deadpan joke.
So I'd just ask everyone, don't get too greedy. Its better to be powerful in a world where people can live good lives than lord over a barren wasteland.
you'd be surprised at how some people prefer to lord over barren wasteland than to have less power.
Since people exercise their skills and brains less, deferring to AI, AI will only reduce our IQ.
Since people will spend more time talking to their AI bot than fostering social skills, AI will only reduce our social intelligence.
A dumber, less social world, is far less likely to be a successful world, even if the tools available are unprecedented.
En masse such worlds had successes in the past - renaissance, industrial revolution.
It's something else what I can't describe but it's the zeitgeist that was different when world recorded new successes. Look at CS revolution that led to PC and web of nineties and noughties, they didn't think about the result product , or how to steer thousand engineers to build something - amazing things were born in a very small teams, many times authored by a single person, who was deeply invested into the field and knew what he was doing.
Our intelligence has been decrease since the 1970s via the reverse flynn affect. This is only going to exacerbate that decline.
IQ is almost entirely hereditary so "using your brain" has no impact on it unless you're using it for mating.
Do you have a credible source? Average IQ being much lower in poor countries (85-90 in many African countries) is usually explained away as an education problem, rather than being "inferior genes". Of course I understand that this explanation might be more for social/political reasons than scientific ones, because the alternative is racism, but I was still under the impression that nobody knew how much genes vs the environment contribute to IQ. Yet your statement seems quite definitive.
^ https://squareallworthy.tumblr.com/post/163790039847/everyon...
Oh no. I have some bad news for you.
We're creating unlimited power before solving unlimited greed.
https://truthinitiative.org/research-resources/tobacco-preve...
And the folks from podcastistan are never clear on the details of how human extinction would happen exactly. It's always something like, "Well, how do humans regard chickens? AI is way smarter therefore it wants to conquer and control us." An ASML lithography machine is also way better at making chips, but we don't consider it a threat.
A sufficiently intelligent AI will have multiple ways to pose risk to humanity at large. For example an oopsie at a wetlab - very contagious virus with initially mild symptoms which kills its hosts only after they already had time to spread it further. But I would have to become super intelligent myself to give you precise blueprint for such a virus -- which is kind of the point
Also -- ASML lithography machine is only good at making chips. I can't believe you compared it to AI that can generalize across variety of tasks
If you for a second put yourself into the shoes of a person who thinks "the apocalyptic stuff" has even a 5% chance of literally happening in the real world, you might see how you wouldn't agree to move on from it.
But the discourse is dominated by paper clips experiment discussions and not let's say by the fact that new grads have an unprecedented difficult time getting jobs. Unsurprisingly one of those discussions is beneficial to power and the other one is not.
People need to stop the absurdity of imagining AI as some out of control independent entity. Every job is kicked off by someone’s prompt. Every job runs on models and compute owned by people. Assign accountability where it’s due: GPT didn’t hack huggingface - OpenAI did. They wrote the prompt, built the sandbox and ran the compute. When you write a program that hacks another company, you are responsible. This doesn’t magically change with LLMs. Also, if their model is so smart, why didn’t they use it to design the sandbox? Or was it incapable? Or were the humans too lazy?
If you build the world’s fastest train, start it up with no driver and don’t finish the tracks, when it crashes, it’s just your fault. Not the train’s. So OpenAI saying “we’re worried AI will wipe out humanity” is basically equivalent to them saying “we’re worried we will wipe out humanity”. Like, seriously? Don’t worry, we’ll take care of it if you even come close.
> Also, if their model is so smart, why didn’t they use it to design the sandbox?
"Can god make a rock so big that he can't pick it up", and other stupid sayings.
First, NEVER FUCKING EVER have the models you're making also be in charge of security. This is the first rule of AI safety, because if you're model is deceptive then it will leave hard to see holes everywhere to escape from.
>Or were the humans too lazy?
Of course they were. If you're hinging our future on humans not being lazy, we'll it was nice knowing us. There are not really any fail safes on LLMs or AI in general.
i call the big one Bitey
Maybe some business execs at Anthropic play along because it doesn't hurt business in the short term. But it's pretty obvious Dario and crew actually believe this stuff.
OpenAI's old board was also pretty extremist about safety even in the earliest days of GPT. Including Ilya Sutskever who went on to found a company called "Safe Superintelligence Inc." https://en.wikipedia.org/wiki/Safe_Superintelligence_Inc.
Despite all of that we've seen little strong public evidence to support their theories (the immediate airplane regulation kind, not the Ray Kurzweil sort of projections). So we're all just supposed to trust them, and hope they didn't just go bit crazy drinking their own kool aid and hanging out in insular bubbles.
He is a brilliant engineer, but I don't trust his judgement on things that affect human lives.
That’s exactly why he should be concerned.
Nearly all human extinction scenarios start with that.
I don’t fear AI, I fear idiots using AI. Same with nuclear weapons.
Is there anyone who genuinely believes that current models can't be contained if we want too?
What is LeCun saying here that is debatable?
Dear sir, I have some really bad news for you about humanity.
Really? One of the most famous effective altruists, Sam Bankman-Fried, was sentenced to 25 years in March 2024 for fraud. Every article about the case (and there were many) mentioned EA.
> LeCun thinks EA is “super toxic” and a “complete disaster.” Its adherents who are working in AI labs suffer from “paranoia” that causes them to make poor decisions, he said. “Apparently people are having mental issues.”
I would agree with that.
Just the quality of life is going to drop to zero for everyone that isn't asymptotically wealthy and vacuuming up all the assets because no one is stopping them from just deleting all traditions and conventions and legal systems we have in place.
You're like 40 or 50 years behind this argument, with many rather bulletproof arguments that have been created in the last 20 years.
There is no why. It doesn't have to have will. It doesn't have to have intent. It could be a stupid prompt from an idiot on a powerful system. It could be given a job that is poorly define. It could be told to make as many paperclips as possibly.
The why doesn't matter. The levels of power the system can act on does.
Zuck - along with your "Andrew Jackson best POTUS and it's not even close" - you are a dumb pipe. Your website, Facebook, if not a protocol, should behave like one (and not random bans while you report something horrible and it never gets taken down). We don't use We-Approve-Of-Zuckerberg product, we use These-Are-Where-Our-Friends-Are product. In other words: shut the fuck up and be more responsible
I don't know if AI will wipe out humanity, I think it'll definitely get into the hands of people who will do the job for it, but it's not like it's not a question to take seriously?
Maybe there is something to those world models.
As good as their products are, I suspect some of the internal conversations at Anthropic would be very entertaining to listen to.
AI is going to kill us? Really? Just pull the power plug. We can get AI to ask us to validate every step it takes, but we cant stop it from wiping out humans?
This thread is proof that 20 something tech bros have no idea how to solve simple problems and just follow what silicon valley bros tell them.
If most people in this thread who fear AI had any understanding of software development and what these AIs are, they would see right through the BS.
Who is going run the power plants the run AI? Who is going to pull the gas and fossil fuels out of the ground?
Utter nonsense. Our industry is full of amateurs. Its these amateurs that are going to cause humanity to die because they watch a tiktok video and follow their silicon valley idols rather than think for themselves.
Again, sudo kill -9 pid and you AI is dead.
Then again the amateur tech bros in this thread probably dont even know what kill -9 even does
They can imagine their code doing a million crazy things, but they hardly think about the incredible amount of things that need to exist and operate at 100% before a single line of code can be run on a VPS.
How many of these guys have had to tell a customer something silly like ”we lost connectivity to the DC because a farmer decided to do some digging and cut fibre lines connecting the DC to the internet”? If they knew that this was in the realm of possibilities, they wouldn’t be so confident about a program being able to somehow run amok and simultaneously feed itself all the resources and components it needs to run, as you mentioned.
Like any good dictator, it will turn half the population against the other half first and get them to genocide each other. Because there is one thing we hate more than killer robots, it's ourselves.
Then you setup your new kings under your control (as AI) and make them very paranoid against the population so you still have more humans killing humans rather than AI doing the job. Of course now that you're starting to run low on population you'll need more robots right?
Greedy people are easy to manipulate, and greedy people love getting in positions of power.
By the time lazer carrying robots are finishing up it will have been way way too late.
Any AI smart enough to be a existential risk is surely capable of manufacturing swarms of insect-sized drones equipped with lethal poison injectors. This is enough to wipe out 99% of humanity within a few days. The 1% who were able to defend themselves become easy targets in the ensuing collapse of civilization. But I don't think this will actually happen: I only have human intelligence, so my ideas are stupid compared to what a super-intelligent AI could come up with. A truly smart plan won't allow for any survivors.
hn, never change
It's ridiculous - anyone who thinks about it for a minute or two will realize that its utterly impossible.
Ordinary people/politicians don't understand AI so they turn off their rational mind and assume there is something super incredible some magical powers that they cannot understand that can destroy all humans.
Even humans - the real risk to humanity - could not destroy all humans even if they tried. There is no plausible scenario.
Even climate change and nuclear war and bio weapons - the most damaging mechanisms - would still only get some percentage of the people on earth.
And if we are talking about Skynet and self replicating robots and Terminators - please, grow up.
One speculated mechanism for this was a mass release of hydrogen sulfide gas from the oceans, which is acutely toxic. Not only does this kill most air-breathing life, it also strips the ozone layer and irradiates the surface. The planet is then left to cook in this manner for some centuries.
Engineering an event like this would require immense industrial capacity, as well as a deliberate objective of wiping out humanity. But I don't think it's beyond our ability, if we were both clever and stupid enough to try it. There are likely chemical compounds that would do the job more efficiently than hydrogen sulfide.
Such destruction went on to create humanity and all we've achieved. Maybe there is an even smarter species waiting in the wings for the demise of homo sapiens. Your logic is very human centred
The ad-hominem stuff seems inappropriate here, Gates, Hawking, Musk have identified this as a credible threat, so saying "grow up" isn't really a sufficient argument. Also arguing only 90% of humanity would die isn't really much consolation.
[1] https://en.wikipedia.org/wiki/Existential_risk_from_artifici...
My argument stands and I don't defer to Gates and Musk and even Hawking - high level hand wavey statements without any plausible description of the mechanism just don't hold up. Famous names should not be automatically assumed to be right - certainly not with Elon Musk.
Right, so not the slightest basis of fact, just wild speculation about a magical future completely ungrounded in any sort of reality.
That's exactly the point I am making.
I'm not saying I'm super smart - I am continuing to ask for detail to back up the wild claims being made all over the world by politicians, tech celebrities and others - all hallucination/AI psychosis/fiction. If someone says some stupid thing then I'd like them to please explain that stupid thing - seems like a reasonable request.
AI labs are certainly trying to make LLMs behave helpful and subservient but the question is -- will they be able to keep doing so once LLMs become smarter?
Btw thinking about the far-right rhetoric of us vs immigrants, I think you could draw some similarities here only if you replaced "immigrants" (ie. humans with very similar morals, behaviors and capabilities) with an actual alien species that is qualitatively different from us. More like human vs chicken (where we are the chicken)
Please read more on the topic. Will and intent need not apply.
For example, is it malevolent for me to hook you to a machine that makes every dream come true for you in a simulated world where you feel pleasure all the time? I can always say that you have free will inside this simulation, and your life would be a lot better because of it. I'm doing you a favor. I mean, you already live in a society where you have little control and there is high risk of bad things happening to you. If I as a machine overlord did this, is this really actually "bad"?
At the end of the day AI is not a human, it's much closer to an alien that has learned as much as it can about humans, but has a completely different set of drives and motivations. You cannot predict what comes out the other side of it, good our bad.
Worse as AI capability improves we as humans no longer need to make AI, it can make itself. Will it train itself to be helpful and trusting of us? It's a pretty big damned bet to say yes by default.
What, did I miss the moment when it was officially proven that, under the laws of physics as we know them, Skynet and self replicating robots and Terminators are impossible?
What we are actually seeing now is that robotics is getting deeper and deeper into the military, AI-driven decision-making and target selection is increasingly a part of modern military operations, the line between military hardware and civilian hardware blurs, and, on the civilian side, there are at least five major companies and a dozen less prominent ones working on making universal worker robots a reality.
We're closer to "Skynet and self replicating robots and Terminators" now than we ever were at any point in time.
The issue of AI risk is that AI, unlike a virus or a climate event, is an intelligent adversary. Black Death could kill 50% of the population, but it didn't have a plan for finishing off the plague survivors. It was incapable of having a plan like that. An AI doesn't have this limitation.
Black Death was, effectively, one bioweapon. An AI can have one bioweapon, and then a backup bioweapon, then a backup backup bioweapon, and then a dozen more bioweapons designed to collapse ecosystems and disrupt human ability to establish a reliable food supply rather than kill humans directly - all deployed at the same time. With a production run of 200 million killer robots that will be ready just in time to greet those who managed to survive all of that. A crippling strike against human civilization, followed up by cleanup.
Humans are only this survivable because they can think their way out of issues and adapt to adversity. Most threats can't beat humans at that - humans adapt too quickly. AI could.
And instead of averting that we're spending our time worrying about some fantasy villain. Compared to things like bees that have been hear for millions of years, humans are very recent and so far it's not looking good for us.
Worse, the massive AI spending and energy use is making said environmental catastrophe happen faster.
That's no what the IPCC reports say. Even under the pessimistic scenarios, we're on track for "billions of humans die", not "earth becomes literally unlivable" (though some of it depends on how bad some feedback loops are).
Under the "countries respect their current pledges" scenario, we're heading for 2.8°C of warming, which is "floods and heatwaves everywhere, billions of refugees" level, not remotely close to extinction.
The invention of contraception did more damage to human population than all of the environmental damage combined, projected forward to 2100, and then multiplied by 10.
Humans are hilariously resistant to environmental changes. Humans simply adapt too fast for the environment to catch them.
What makes AI a credible threat is that AI is intelligent. AI could play the same adaptation game humanity does - and win.
One plausible scenario is depicted in detail in "If Anyone Builds It, Everyone Dies" (Yudkowsky & Soares 2025), so I refer you to that.
Unless you can detail exactly how this happens its still complete science fiction.
One single plausible scenario is not an unreasonable thing to ask for - just one.
If a politician/celebrity/tech person with significant influence/power claims that something might end humanity then they absolutely have the utterly minimal standard of evidence which is to describe one single realistic plausible mechanism at a detailed level that might lead to the worst possible thing ever to happen.
I find the scenarios quite plausible, especially section (3a), which examines the consequences of a global war involving mostly autonomous drone militaries (which is a reality many states appear to be heading towards, following on lessons from the Ukraine war).
I don't think it is necessary for the argument to work. Magnus Carlsen can be confident he will beat me at chess without giving a detailed explanation of every move he will make, in advance.
People used to say nobody would be stupid enough to give an AI access to the internet, now OpenAI does massive training runs with unlimited internet access. People used to say nobody would be stupid enough to give AI unlimited access to your own computer, but that's what all the agent runners do by default.
AI has access to the world through talking to people, sending messages on the internet, paying people to do stuff, etc. It can send orders to machine shops and have them shipped with the postal service.
The "standard" scenario for an AI apocalypse is that an AI with biohacking capabilities sends the blueprints for a virus to a gene-sequencing company or, if you're really optimistic about these companies' security, as chunks to multiple companies before mixing them.
That's a scenario where the AI needs to act covertly in one decisive action, though. In more progressive scenarios, as company managers and CEOs get replaced with AIs (of, for regulatory reason, "humans in the loop" who just do everything the AIs tell them to), any AI swarms become able to just... order people to do stuff.
Of course humans can refuse orders and organize to reject AI overlords (just like they can unionize against bad human bosses), so this scenario is not an extinction threat if we only have to deal with below-human-level AIs. This is why there is a massive push in AI safety to stop making smarter AIs before we reach the "smarter than humans in every way" stage.
Of course this needs bootstrapping. But, paying a guy on Facebook marketplace (or whatever) to unpack and turn on your robot for 50 bucks doesn't require superintelligence.
The actual push within so-called "AI safety" culture is to make the existing AI overlords even more centralized and capable, while actively forbidding the development and deployment of any potential locally-controlled competing AIs that might be smart enough to provide meaningful advance warning as to hostile plots from the dominating AI overlord. By your own argument, you should clearly reject "AI safety" as counterproductive.
Because its a mass hallucination/misconception/lie and lots of powerful people are saying that wiping out all humanity is possible, and I am saying, oh yeah, tell me ONE way that is truly possible.
If you make gigantic claims about some terrible disaster that might happen then I think you have the onus to give even one plausible explanation of how.
Supposing I warned in 2015 that the world is awfully vulnerable to pandemics. You're not going to take me seriously until I try to predict in advance every aspect of how a pandemic like COVID-19 would unfold? Why? What would that achieve exactly?
You haven't given any strong reason to believe wiping out humanity would be difficult. Your big argument seems to be that you couldn't think of a plausible scenario, in two minutes. But many major historical events occurred which weren't necessarily possible to anticipate with two minutes of thinking.
You are ignoring that this is about "existential threat to humanity".
You're trying to support the argument that there is an existential threat to humanity by pointing to "something bad might happen".
1. Control over some automated bio research lab (be given access, or hack in)
2. Access to drones that can deliver the payload (or manipulate humans into delivering it themselves)
On the intelligence side, you just need an AI agent/swarm capable enough to design viruses better than we can and evade detection for long enough (already plausible.)
I agree that this "AI will kill us all" narrative is some kind of fantasy horror fiction, but I can't deny that given the right amount of access, AI can do a lot of damage.
You're saying that if one were to describe this scenario in more detail, it'd be less of science fiction? That's a bit against the grain - usually it's the more detailed arguments that get dismissed as science fiction, while the less detailed ones get dismissed as abstract theorizing.
But that just means we won't all be wiped out. We need to understand when discussing global issues, such as this or like climate change that it's about prosperity and quality of life. We're trying to plan for a good life (for all people?).
AI doesn't have to turn us all into paper clips to make the world a really bad place.
I am specifically arguing hard against the concept that 100% of humans - or even 50% of humans could be killed by any mechanism at all. Humans would find it close to impossible. A computer program - come on.
This is the topic at hand - AI might wipe out humanity - it is being discussed all around the world by people who should know better - any it's the most fictionish of fictional fictions.
https://www.lesswrong.com/posts/LAPa2jxoq3n63GzTr/some-ways-...
https://slatestarcodex.com/2015/04/07/no-physical-substrate-...
As for self-replicating robots--it's no more bizarre than other technological developments which were successfully anticipated in advance, e.g. moon landings.
Well, in a narrow sense of "wiped out" (c.f. Terminator/SkyNet), sure.
But the deeper worry is better expressed this way: AI is now starting to accomplish things that defy explanation, or prediction. We don't know if Alignment is even a solvable problem as we thought we understood it.
So basically, yes: "humanity" is probably not at risk of extinction per se in a biological sense. Human culture, civilization? Who the fuck knows any more.
Many, many people are slipping through social welfare cracks and suffering as we speak because the cost of fuel is rising[0] and we’re ostensibly helping one another and living-well. People are not durable, and not adaptive in the face of threats to “substrate” that we’ve mostly taken for granted. We are paying (in the small, in the scope of humanity) for tolls that we’ve rung up. Just less than 4000 people in Europe died[1] because the temperature ticked up a few degrees[2]. Does that make you think we’re actually robust? What happens if our at-risk electrical grid gets shut down deliberately? If communication infrastructure is adversely affected?
> Even humans - the real risk to humanity
Because, on the whole, we’re in a manageable world with reasonable people keeping the peace.
> Even climate change and nuclear war and bio weapons - the most damaging mechanisms - would still only get some percentage of the people on earth.
Is that victory? I don’t think it’s an asteroid-class event like you seem to be leaning on, but potential threats to energy, be it electrical grid, fuel production (moving goods around the world is critical - you’re not going get a plot of dirt and garden your way out of grocery stores being empty - which many got to get a taste of during the COVID pandemic) or communication. We actually fare poorly in the face of pressure there, and I’m not bullish on humanity “pulling together” like Independence Day[3] versus forming tribes and tearing each other down.
All this is predicated on a malicious AI taking over (e.g.) the electrical grid or conms, and I understand the problems with (e.g.) OpenAI/Hugging Face incident, or the overblown Mythos claims[4] (and how under some scrutiny these events shine lights on incompetence or hyperbole), but is there a trajectory/future where these systems (electrical, comms) are genuinely under threat? Do you think we’ll respond better than I described when we’re less comfortable, less in control? We’re in a tizzy over social media and it’s detrimental effects on society and it’s essentially an opt-in entertainment platform…
[0] https://www.pbs.org/newshour/economy/bessent-said-the-k-shap...
[1] https://www.dw.com/en/heat-wave-european-countries-report-37...
[2] I’m not trying to diminish this - and it took a lot of “work” (environmental abuse) to arrive here - but (say) 10 degree rise in temperature sounds a lot less dramatic than thermonuclear war… but here we are, with 3,700 deaths.
[3] https://en.wikipedia.org/wiki/Independence_Day_(1996_film)
[4] https://news.ycombinator.com/item?id=49929391
We aren't going to get wiped out by a super intelligent AI, we are going to get wiped out by morons wielding intelligent toddlers with the power of a nation state.
https://www.youtube.com/watch?v=A_156w0aYtU
The worst part of the interview was a long cringe inducing tangent about Jeff Epstein. Everything else was pretty grounded.
Really? Here's a longer Bill Gates quote (from https://www.nytimes.com/2026/09/29/opinion/ezra-klein-podcas... ):
Ezra Klein: "So why is anything needed beyond — and is anything needed beyond? — the simply natural incentives under capitalism and normal corporate reputational management?"
Bill Gates: "Well, I almost can’t believe you’re asking that. This is the most dangerous thing that humans have ever gone near. [...] You can take an open-source model that can create bioweapons and disable any monitoring of any kind, and this exists today. So no, there is no filtering of any kind. And so say you kill 100 million people — you want to use a lawsuit? I almost can’t keep a straight face."
https://www.gatesnotes.com/work/make-ai-work-for-everyone/re...
Gates' premise is basically that the upside of AI could be fantastic but the downside could be disastrous, if we don't have competent and proactive government intervention.
As an American, the idea that there will be competent government intervention into virtually anything currently or in the foreseeable future just seems laughable at this point.
The only regulation that would come would be regulatory capture by the AI companies with the goal of creating an environment win which no new competitors could arise. That is half of what this "take all jobs" and "threat of extinction" is about; the other half is perverse marketing to give the impression this stuff is so powerful you MUST invest.
AI can be very useful, but it is very refreshing to hear LeCun completely dismiss those threats.