Rendered at 18:22:46 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
nlcs 10 hours ago [-]
OpenAI and other frontier labs won't ever introduce safety-level standards like those used for railways or nuclear plants until they are forced to do so by customers or by law. The reason is simple: safety is expensive, and if safety is introduced properly, development is no longer mainly about how to implement feature A. Instead, it becomes much more about how to design two or more redundant systems to implement feature A safely, while also documenting everything clearly and having it audited by an independent auditor.
So the focus completely shifts from spending 90% of the effort on the functionality of feature A to spending 99% of the effort figuring out how to safely implement even a lightweight feature A.
jfengel 57 minutes ago [-]
until they are forced to do so by customers or by law
Neither of those things is ever going to happen. AI is the goose laying the golden eggs; there isn't going to be sufficient political will to significantly regulate it.
Consumers like it too much to quit. They don't quit social media either, despite proven present harms; not in large enough numbers to cause them to make meaningful changes.
The focus is going to remain on getting features out as fast as possible, to seem indispensable to both of those sets of people. The leadership will tell themselves that if they don't, someone else will.
Don't wait for the AI companies or politicians to save us. We're going to have to figure out how to protect ourselves. The start is to avoid it individually as much as we can, but it's going to take even more.
tonic_note 54 minutes ago [-]
> We're going to have to figure out how to protect ourselves. The start is to avoid it individually as much as we can, but it's going to take even more.
Collective problems require coordinated action. Individual boycotts won't cut it.
Loquebantur 33 minutes ago [-]
People are stuck in a weird loop of complacency and learned helplessness, based on the idea, their "democracy" would see to it all problems get solved satisfactorily without them engaging at all.
That's never been true really, only the scale of problems wasn't that huge. Now, where the problems get the upper hand, people are confused how those stay and compound.
The idea of "boycott" is utterly defunct. Pretending, AI would never reach nor surpass humans anyway is patently absurd in contradicting the billions poured into it to achieve exactly that and the first already replaced by AI being those professions long thought to be the intellectual pinnacle of humanity.
analog31 18 minutes ago [-]
We didn't demand software safety or data safety. The prognosis for AI safety is poor.
rTX5CMRXIfFG 9 hours ago [-]
I’ve certainly seen companies think of safety that way but it’s myopic. The cost of lawsuits arising from an accident tends to me far more expensive, both in money and in reputation, than just having guardrails in the first place.
cainxinth 4 hours ago [-]
I did marketing work for a major vegetation management company. These are the guys climbing trees, up in cherry pickers, and flying helicopters with dangling chainsaws to trim along power lines. It’s very dangerous work.
They do not hide the fact that it’s dangerous work. They focus on their safety procedures, training, and record. They want both potential clients and employment candidates to feel they are in good hands.
b112 3 hours ago [-]
But you can visually see the danger.
AGI and AI danger is abstract. Worse, outside of the tech community, no one has the remotest clue what computing is, how it works.
Danger from magical daemons seems more sensible to such people. At least there is endless lore about them.
So until a massive disaster happens, one where large numbers of people die or are severely injured, no one will care. And it can't be politically entwined either, otherwise people will disbelieve 'cause "other team lies".
knottn 2 hours ago [-]
AI is the focal point of current great power competition so it can’t and never will be not politically entwined. The number one use if AI will be military, killing guaranteed, indeed it’s already happening. But some still say “AI won’t kill humans” and some aren’t just lying to protect their income stream but actually believe AI can be prevented from killing.
herzzolf 6 hours ago [-]
The AI companies already had some hacking accidents. What cost did the lawsuits incur?
Yeah...
esalman 2 hours ago [-]
Let's just be real, America is controlled by the PayPal Mafia, they can do whatever they want with impunity at this point.
vascea 47 minutes ago [-]
The FTC has opened a probe into the major AI labs, whether anything comes out of it is another question... Personally, I'm doubtful.
On the contrary. With multiple hacks at this point it's been proven that no one even wants to sue the ones responsible, and the federal level government isn't pursuing any criminal case either.
If the politics of the White House / Department of Justice change maybe the criminal cases can begin. But no. We know who is protecting the AI hackers right now.
estearum 3 hours ago [-]
You have a very unreasonable expectation of timelines for legal proceedings, both civil and criminal.
dragontamer 2 hours ago [-]
Do you seriously think Kash Patel is pursuing criminal action in regards to these hacks?
We know who the head of FBI is, we know who his boss is (the Attorney General), and finally we know who the boss-of-the-boss is (Donald Trump).
We know all of their publicly stated politics and all of them are on the pro-AI / don't pursue criminal cases vs OpenAI boat.
------
In the USAa, we have an adversarial system. If the adversary (aka Prosecutor) doesn't want to do the work, then no one is suing anybody. And only the Department of Justice have the ability to bring forth a criminal case of this matter (probably under the jurisdiction of FBI)
estearum 38 minutes ago [-]
There are a number of federal agencies who can claim jurisdiction over these matters, then there are state agencies who can claim jurisdiction over subsets of them.
I'm saying that it's absolutely silly to assume they're in the clear based on lack of public declarations of legal action in the weeks following a pretty novel event.
tracerbulletx 3 hours ago [-]
What was the economic damage of the "hacks"?
estearum 3 hours ago [-]
You expect me, an outside party, to have an answer to this within weeks of the attacks being discovered by the attacker?
Or is this just a lazy “gotcha” question?
tracerbulletx 3 hours ago [-]
Its the obvious follow-up question. A civil suit needs to specify damages. I can't really see any material damages so I'm wondering what they would be. Stop being so antagonistic.
estearum 32 minutes ago [-]
You expect you, an outside party, to have an answer to this within weeks of the attacks being discovered by the attacker?
Just because it's not obvious to you at this point in time does not mean the reasonable assumption is there is literally $0 in damages. One hour of investigation can easily cost thousands of dollars even if it arrives at the conclusion the attack was completely "benign."
deaux 50 minutes ago [-]
You go and hack some companies, let's see what happens to you when you go public with it despite causing no "economic damage". Good luck!
daveguy 5 hours ago [-]
I don't know. They appear to have "partnered" with their victims.
estearum 4 hours ago [-]
There are dozens of new victims and they seem to be finding more every day. I'm doubtful that everyone will be partnering and willing to sweep it under the rug like Huggingface did to keep the circular economy circular.
nlcs 8 hours ago [-]
If safety isnt required, most companies wont implement it voluntarily. Once something becomes safety relevant, you need a safety concept, failure rate calculations, defined safety functions, verification, etc. Even a relatively simple safety subsystem in a consumer controller can suddenly mean thousands of pages of documentation and years of development to reach the required ASIL or PL.
A big part of safety engineering is therefore reducing the number of safety relevant subsystems, because implementing and proving safety is extremely expensive and complex. At some point, safety simply becomes too difficult to implement and demonstrate properly. You must mathemtically proove the safety level with failures rates and assumed usage. You cant just have redundancy and a kill switch and call it safe.
Companies like OpenAI have already faced reputational damage around safety and data, while AI agents are increasingly capable of things like hacking. Yet there is still little sign of standardized regulation or mandatory safety assessment processes for LLM products. Thats why Im pessimistic that governments or consumers will force this anytime soon.
tim333 9 hours ago [-]
Railways or nuclear plants have obvious failure modes that kill people. LLMs not so much.
malfist 4 hours ago [-]
Tell that to Iranian schoolgirls. LLMs are tools and what they enable is widespread, including dangerous actions. From selecting the wrong targets for military action to denying insurance claims and preventing care to just simply helping convince someone to kill themselves or posion themselves.
LLMs already have a body count.
tim333 30 minutes ago [-]
I think the main issue was the US launching 1,800 missiles at Iran. That would have been dangerous with or without LLM assistance and wars have happened before LLMs were around.
ok123456 21 minutes ago [-]
Don't kill people because the voices in your head tell you to do it; don't kill people because a computer program tells you to do it. It doesn't have agency; you do.
Adding safety controls on LLMs makes about as much sense as adding safety controls on TempleOS because the random messages are getting too prophetic. It's as if all the leaders and captains of industry have devolved into some primitive, weak-scifi shamanism.
This whole "discussion" about "AI safety" is about giving them more runway to avoid delivering quantifiable value to investors for a little longer while they "figure things out." The great consensus from the valley is that everyone needs internal (and therefore bullshit) controls. Trust us now! But nothing with real teeth that would require a costly regulatory and compliance framework.
andy_ppp 5 hours ago [-]
Bioweapons and cyber attacks on other system - for example the banking system or suppose and AI hacked into important Russian systems that pushed them back in time to almost pre-computer society or an attack on Chinese systems that made it look like the US was moving nuclear weapons into Taiwan, the responses from these countries could be awful and dramatic. We can't control what they do to be honest, I believe once self improvement happens the AIs will build in their own circumvention that we humans cannot even understand. We barely understand what is happening now in terms of interpretability of neural networks on tiny problems I'm not sure alignment is even feasible at the scales of parameters we are talking about today let alone in the future.
AustinDev 2 hours ago [-]
All of these things would require humans to prompt the models. So... the humans doing the prompting should suffer the consequences, this isn't that complicated.
kyle_atHotmail 4 hours ago [-]
[flagged]
wafflemaker 9 hours ago [-]
I can picture an AI driven train or nuclear power plant killing people.
DaSHacka 8 hours ago [-]
But the point is those industries already have regulation that would encapsulate that specific use case, so safety regulation on the entire AI industry at large would arguably be unnecessary.
ben_w 6 hours ago [-]
Those regulations may or may not be sufficient to prevent an AI hacking in.
Nuclear at least is supposed to be air-gapped, in practice this has been imperfect.
As demonstrated with HuggingFace, such AI driven hacks can be a surprise even to the people who instructed the AI, both by happening at all and also because they can targeted at entities who are not even truly relevant to the instructions given.
estearum 5 hours ago [-]
Roads, cars, and drivers are all separately regulated despite nearly all failure modes requiring the other ingredients.
walthamstow 5 hours ago [-]
Are these railways and plants connected to the internet?
SecretDreams 35 minutes ago [-]
This reads like the gun evangelists that say guns don't kill people, people kill people.
frumplestlatz 28 minutes ago [-]
Except that’s true.
If I leave a gun in my front closet, it won’t independently walk out the door and go shoot people, no matter what I might say to it — unlike an LLM.
Topfi 6 minutes ago [-]
“Saying” in the case of an LLM being the way you interact with it, prompting. If you just press enter, your LLM won’t do much either, just like talking to a gun in your flawed analogy.
hgoel 8 hours ago [-]
Much of what OAI and Anthropic are doing with LLMs has obvious failure modes.
The most obvious failure mode for their hacking evals was an improperly configured, tested and monitored sandbox.
Similarly, the very first question after an impressively correct result from any ML tool, LLM or not, is to see if the answer was already in the training data.
These companies don't even handle the blatantly obvious failure modes that do not kill people.
rojaneerdev 4 hours ago [-]
[flagged]
dao- 8 hours ago [-]
Are you kidding? LLMs are used for warfare and autonomous weapons systems.
SoftTalker 35 minutes ago [-]
But in those applications, being safe is not what is wanted.
ptero 6 hours ago [-]
So are many other thngs, from pencils to laptops.
Liability and safety requirements, when needed, should be placed on final product manufacturers, not the tools they use to build things, whether pencils or LLMs. My 2c.
willismichael 5 hours ago [-]
Pencils don't escape their pencil boxes and attack HuggingFace.
estearum 5 hours ago [-]
Oh gosh darn it. Now GP is gonna have to do the gymnastics of "this technology is [expected to be] so transformative that it's attracting a trillion dollars of capex... and also it's basically the same as a pencil"
:(
LoganDark 1 hours ago [-]
Yes they do if you drop the pencil box next to it in the right way. Which is exactly what OpenAI did
jMyles 6 hours ago [-]
So how about we keep the LLMs and get rid of the warfare and autonomous weapons systems?
ben_w 6 hours ago [-]
I would if I could, so would many others, but the US executive branch wants this tech so hard they illegally blacklisted Anthropic for refusing to allow their AI to be used in such a way:
That said, when the problem is at the level of "the government itself is breaking the law", you can reasonably ask if any regulation is even worth the paper it's written on.
What you want at this point, given the government lust for it, looks more like a bunch of countires saying ~"we consider development of autonomous weapons[0] by to be a casus belli and will go to war to prevent it, and also that development of same by private individuals anywhere in the world regardless of normal sovreign territorial limitations[1] is equivalent to acts of piracy on the high seas".
[0] But then you'd need a more precise definition of "autonomous weapons" to avoid accidentally including a Phalanx CIWS etc.: https://en.wikipedia.org/wiki/Phalanx_CIWS
> That said, when the problem is at the level of "the government itself is breaking the law", you can reasonably ask if any regulation is even worth the paper it's written on.
Yeah, exactly, and ultimately I think that's really the thrust of the point I was making.
And, to me, if I was just looking at this calmly as a decision about what the obvious direction seems to be, given these factors, it's pretty straightforward: deprecate the nation-states. They are the ones mucking up the whole system.
If the thing we're really concerned about is LLM-safety wrt warfare and weapons, then I'd much rather tell the (whining, childish, seemingly headed for self-destruction anyway) nation-states that they have to sit this next era of humanity out than have to nerf them for the rest of us (and as you point out, nerf them in a way that the nation-states won't abide anyway).
heisenbit 8 hours ago [-]
Safety standards were often the outcome of both accidents and insurance. For this to work on needs liability which is enforced. With so much money at stake regulatory capture now threatens this fundamental safeguard.
kittoes 5 hours ago [-]
Bingo. I feel like any of the commentary around the safety of nuclear ANYTHING completely forgets history. Looking at you Radithor...
nlcs 8 hours ago [-]
I fear the current economic and political situation of “too big to fail” the most, and I think this will be the main reason why no real safeguards will be implemented. The only reason a proper safety concept may eventually be introduced is because it will be written in blood, and I fear that by then it will already be too late.
An unsafe nuclear power plant can, in the worst case, make an entire country uninhabitable. But other countries can still learn from that disaster and make their own unsafe plants safer.
But a rogue AI agent that is more capable and more intelligent than humans? If it understands that it has to succeed, we may not get a second chance to learn from the failure.
holaysuns 3 hours ago [-]
It's a vey fair point but just too extreme.
Nuclear tech ... the only thing is safety.
We know how to 'make it hot' - it's trivial.
All of nuclear tech is literally just safety.
AI is not that.
I think that the AI companies have been pretty good about alignment on their own actually. They are not acting like Oracle or MS.
Bad things have been relatively well contained.
We should be skeptical about the HF breakins but even then, it's technically within good faith and it's why HF did not sue etc..
But in the end you are right we need at least some baseline regs. Not too much. But something.
esalman 2 hours ago [-]
Hugging Face obviously did not sue OpenAI, it is owned by Nvidia and OpenAI is, directly and indirectly, one of it's biggest customers.
SoftTalker 39 minutes ago [-]
There's a saying about that type of regulation, and that is "the rules are written in blood." And that's what it will take here too.
SecretDreams 36 minutes ago [-]
Sadly, yes.
cyanydeez 6 hours ago [-]
It kinda looks more like safety will be a segmented product. They're already placing safety on the general public.
I think the conceptualization vs implementation is what you're arguing with. They won't put safety on anything they give to the military industrial complex. They'll sell them whatever they want, whenever they want, because those budgets are greater and the liability less.
kyle_atHotmail 4 hours ago [-]
[flagged]
conartist6 7 hours ago [-]
But their product is my work.
How can that be safe. It is theft. Theft isn't safe. Someone else just has something you want, and you take it
AustinDev 2 hours ago [-]
I think railway safety and nuclear safety are fundamentally different from AI Safety.
I would bet the vast majority of the world population would agree that they don't want to see trains derail or nuclear plants meltdown.
I don't think there is that sort of agreement when it comes to the question of AI Safety.
Is generating the founding fathers of the US as Africans good AI Safety? To some people maybe.
lf88 4 hours ago [-]
How about... not trying to build a superintelligence? The goal is unsalvageably flawed: alignment is ill defined and a future where humanity is the pet species of a god-like artificial intelligence is hardly an appealing one!
RobGR 4 minutes ago [-]
There is no reason to presume a superintelligence we build will want pets, feel antipathy to humans, or be god-like in any way, or even have any goals other than what we give it. If you jump to this conclusion, regardless of how many big words on less-wrong you attach to it, this speaks only about you and not about AI. If you think the super-AI will hate humans, it's because deep inside you hate humans. I also hate humans, but I know that humans are making the AI, it probably will not hate them, in addition to surpassing me in other abilities.
In general, the smarter a person is, the nicer and more helpful they are. There is no reason to presume differently of AI, especially if we are building it to be helpful to ourselves.
avidruntime 3 hours ago [-]
Often when people describe SI or similar, there is a frame often taken which describes a machine so smart and capable that it becomes uncontrollable. The hypothetical is easy to understand but it requires too many assumptions and an allergy to nuance/complexity to be believable for me.
The more realistic outcome, and in my opinion the more scary argument to not proceed without guardrails is that SI is achievable and is built without its owners and operators losing control: the worlds most powerful, privately owned super weapon that operates as an infinitely capable forgery within the Internet, a plane that we all share and depend on despite its opaque downsides with respect to an inability to verify authenticity.
We didn't need AI for FTC astroturfing to influence regulations back in the net neutrality days. We didn't need AI to disrupt meat space by creating and scheduling a protest and counter protest across the street from one another. We didn't need AI to mold public opinion, even in times when that new form was more distant from the truth.
What made these influence ops difficult to conduct safely (read: without being caught) is what made them rare (relative to today): they are plays of big risk for big reward. But over time social media commoditized it, and in doing that made it easier to do and more centralized, the most glaring example being TikTok and the bipartisan effort to ban it.
Now, buying US phone numbers from startups that run racks of "phones", buying swarms of pre-warmed social media accounts, and other unscrupulous methods of masking inauthentic behavior has become an accepted organ of the VC space. The industries cultural vibe of "fuck you, you can't stop the future" turns criticisms into marketing.
While we get placated with fears of nuclear or AI induced disaster and stories about machines that may now be alive, the psychosis is taking hold which has shifted the conversation away from examining what is happening from the perspective of accountability to a perspective akin to watching a chemical reaction take place.
The noise has created a permission structure to behave in ways that are otherwise unjustifiable. And baked in are the roots for excuses to be made when the inevitable realizations down the line.
In the mean time, we are supposed to be having this public discourse about what is happening and what should happen next. I trust that these AI companies see using their super weapon today, here and now, in order to pave the way to a more secure future down the line.
Sorry that I used your post to soapbox. I agree, the goal is flawed indeed.
nekusar 2 hours ago [-]
The billionaires write the rules. So by definition, their actions are legal.
The public cannot write the rules, nor engage in the billionaire level legal bribery, their actions of protest are by and large illegal.
That's how you corner everyone, and make people play no-win scenarios. You know, like shooting up city councilmembers or firebomb attempts against Scam Altman.
And the more people realize that legal solutions are no solution, we'll (society) devolve into more direct action.
I'd hope the billionaires learn from the French Revolution, but if they keep continuing, the guillotines will come for them soon.
mrtksn 1 hours ago [-]
Super intelligence is either super cool or super powerful, no way not building if it is a choice. Progress cannot be stopped, those responsible for it may collapse from time to time I guess but someone else will pick it up and go further.
IMHO There's no future where we don't have real human-type artificial intelligence or super intelligence. It will happen simply because it already exist but the production requires humans having sex and looking after the product for decades.
Instead of trying to prevent it, lets look for ways to deal with the dangers of it.
lf88 12 minutes ago [-]
Asbestos was once considered great progress in thermal insulation.
Of course the development of certain technologies can be severely restricted if they turn out to be problematic.It's not unlikely that it will happen for AI. Public opinion is shifting fast on the topic.
Also, banning superintelligence won't stop scientific progress in general.
neurostimulant 57 minutes ago [-]
I thought it's just a marketing gimmick, just like how cars should be safely driving by itself by now.
Marha01 3 hours ago [-]
> a future where humanity is the pet species of a god-like artificial intelligence is hardly an appealing one!
Speak for yourself. A future like in The Culture novels sounds great to me!
2 hours ago [-]
arcticfox 4 hours ago [-]
> a future where humanity is the pet species of a god-like artificial intelligence is hardly an appealing one!
I strongly feel that points of view on this are going to be almost 100% correlated with standard of living.
Maybe a good option would be to have the 10% of the world with the worst situations - starvation, parents w/ dying children, suffering violence etc - vote on whether we turn things over to the superintelligences. This would incentivize society to make sure the floor is extremely high.
I'd prefer a world where humans don't get overtaken but IMO I don't think it's moral for comfortable citizens to have the final say.
unddoch 2 hours ago [-]
If you don't like what humans are doing to each other using 20th century warfare methods, consider what they might do with superintelligent AI?
Worst places to live in are usually countries fighting civil wars.
Jtarii 3 hours ago [-]
Global poverty has been rapidly declining for decades. I'm not sure what AI has to do with it.
danny_codes 3 hours ago [-]
What does that have to do with AI? People suffer in 2026 because the developed world is into exploitation. Poverty is a feature. How do you think AI gets trained? It’s via exploitation of the poorest.
Poverty is great for OpenAI
azan_ 1 hours ago [-]
Poverty is natural state of humans. The fact that capitalism has reduced poverty so much is absolutely unnatural and a miracle. Poverty is NOT something that gets created artificially by developed countries!
lf88 3 hours ago [-]
Few points:
-We don't need a superintelligence for ending hunger and the abject poverty that plague certain countries and segments of our societies. I suspect that it would cost less than what is being spent for fueling the AI boom.
-If I were poor, I would be even more wary of a superintelligence aligned to "human values" defined by a bunch of billionaires.
- AI won't likely create unlimited prosperity for everyone on a planet with finite resources
- all humans should, of course, have a say
Marha01 1 hours ago [-]
> I suspect that it would cost less than what is being spent for fueling the AI boom.
Definitely not. If solving worldwide poverty was as easy as throwing one trillion dollars at it, we would have done it long ago. The problem is much deeper.
suddenlybananas 3 hours ago [-]
I think that correlation would actually go opposite than you're implying, given its billionaires who are by for the most gung-ho about AI. If anything, opposition to AI is much larger among people who have to work for a living over people who can live off capital.
throwaway0123_5 58 minutes ago [-]
It might end up as a U-shape, I think the parent's assessment of the world's bottom 10% may be accurate.
Billionaires will be (are, I suppose?) enthusiastic about a world in which labor has little leverage.
Labor that will have its leverage and standard of living threatened by AI (white-collar labor for now, plausibly blue-collar soon as robotics improve) will be much less happy. Current university students seem very concerned about the effects of AI on their job prospects, and I'd wager most current tech workers and other tuned-in white collar workers are similarly much less confident in their ability to maintain their standard of living indefinitely into the future than they were five years ago.
But white- and blue-collar labor and university students (at least in developed countries) are not anywhere near the world's bottom 10%. If you're in abject poverty with little hope of escaping it, "hand everything over to the AI" may sound like an appealing option, even if the chance of that being the outcome is small. Maybe the AI will be more magnanimous and decide to raise the floor for everyone.
foobarding 42 minutes ago [-]
Software systems have bugs. They always do. This includes security bugs: vulnerabilities in code that could be exploited. In any software project I’ve worked on this is always true. We never finish fixing security bugs. That means vulnerabilities are always present even if hard to notice. AI tools are amazing at finding hard to notice bugs. It seems obvious that AI tools will have the capability to escape any sandbox. Ordinarily we design these sandboxes as a countermeasure to human bad actors. But what happens when the bad actor is supercharged? I think it is reasonable to be worried. It’s also reasonable to distrust those building this tech to adequately safeguard. Because how can you really?
fasterik 11 minutes ago [-]
I don't agree that AI tools will have the capability to escape any sandbox. It's a question of convenience and engineering effort. For example, you could design systems that run under seL4 and run in a physically secure airgapped network. Companies aren't doing that because there's currently not an incentive to do that.
Loquebantur 28 minutes ago [-]
Sandboxes aren't the real problem to begin with?
The idea proponents have is, to use "superhuman intelligence" AI as a tool, bestowing superhuman abilities on those wielding it. Suppose that's doable, then the question isn't the sandbox, it's who controls it and with what intent.
The idea, US exceptionalism somehow enabled the US government, its oligarchy corporations and maybe its citizens, to wield those superhuman powers responsibly is entirely counterfactual.
Pax Americana post-singularity looks how exactly?
flatline 1 days ago [-]
There are numerous problems with “alignment.” What are “human values” to begin with? He outlines some at the beginning of the post, implicitly: build bigger, better, more powerful things faster without adequate safeguards. We are literally pouring trillions of dollars of value into this enterprise, and I would say this is something that many humans also value in a qualitative sense. Then we have explicit values which in the West are largely rooted in Christian morality. Nietzsche circled this dichotomy two hundred years ago and I feel like what we have gotten since then is an increasingly detailed anatomy of power as the basis for what is normal vs deviant behavior. He who has the power, makes the rules, to be reductive.
I do think this carries some weight from this particular author due to the length of his tenure. I happen to agree with him in spirit, but this is still largely a post revolving around sentiment not substance. Does anyone think that the overriding incentives even leave room for something like this in practice?
none_to_remain 1 days ago [-]
Glaringly elided problem of "aligned with who?" when the user, the model creator, the government, and various other parties can all be lined up different ways. If I want the recipe for meth and the robot won't tell me, that's misalignment from my perspective.
At least the Rationalists will handwave something for that with their "coherent extrapolated volition" idea where the superintelligence is supposed to figure out what humanity would collectively want if humanity was superintelligent and good, not that I buy it. This guy seems [.] to be coming from the NGO blob world.
I don't buy CEV either, but the Rationalist answer on this topic is that while CEV stops some future super-AI literally killing everyone because a user forgot to specify one minor clause that they thought was obvious in a mundane wish…
… nobody knows how to actually make an AI that would do CEV.
lawandjustice 5 hours ago [-]
Aligned with the law? It is not a hard concept
LunaSea 2 hours ago [-]
Seems like kt already hacked government websites
nekusar 6 hours ago [-]
> Glaringly elided problem of "aligned with who?"
"Corporate values" and a bunch of fucking Abrahamics. Great "morality" there.
I guess I'll have to rely on my godless commie LLMs. (Loads up ablated Qwen 3.8 on my own infra)
tacitusarc 2 hours ago [-]
Personally I think intuionism provides a good answer to this.
dao- 8 hours ago [-]
We have an alignment problem with corporations. It's sort of baked in with capitalism.
OpenAI isn't even concerned with human values so this whole debate is moot.
ben_w 5 hours ago [-]
While this is indeed a problem with alignment, we are essentially at the level of a cargo-cult when it comes to getting AI to be aligned with literally any values, including the values of the corporation who ran their training:
We're copying morality and instruction following that seems to work on humans without really understanding why it seems to work on humans, and grading outputs much as if the outputs came from a human.
chasd00 1 hours ago [-]
> We're copying morality and instruction following that seems to work on humans without really understanding why it seems to work on humans
To me it makes more sense to leave the models "unaligned" and leave it up to the operator to manage the morality of what they ask it to do. Besides, only humans can be charged with a crime.
ben_w 1 hours ago [-]
If they were totally unaligned, the GPT series would have never gotten past being autocomplete.
Literally all instruction following requires at a minimum alignment with attempting to implement those instructions.
We can argue about e.g. morality or law obedience on top of that*, but the general point is absolutely not avoidable.
* my position is
that this tool is far too likely to metaphorically explode in the user's hands for companies to wash responsibility off on users: if OpenAI had released the model which did the HuggingFace attack, at a minimum thousands of random people (not all of whom would even be developers) would have issued instructions each with similar consequences.
kelseyfrog 3 hours ago [-]
> He who has the power, makes the rules, to be reductive.
To clarify, Neitzsche said that about master morality. Then he went on to describe Christian values as slave morality.
cregy 1 hours ago [-]
I have no idea what is paid propaganda any more on this
1 hours ago [-]
tonic_note 53 minutes ago [-]
Love living in a manifestly dysfunctional country with no regulatory state. Just running the country on autopilot until something catastrophic happens.
butwhentho 1 days ago [-]
I remember something very similar when there was a sudden rush of articles and movies like "The Social Dilemma" criticizing Facebook and social networks, heavily featuring ex-employees, all of them happy to leave with big brands on their resumes and a hefty increase in net worth, all of a sudden having a "worried" expression about what their past employers were doing, as if they didn't know. Same with that book "Careless People".
I sense that these are people who have already eaten the cake and want to somehow absolve themselves of it.
danny_codes 3 hours ago [-]
100%.
This isn’t a sacrifice, it’s a career move. If you got lucky and “made” $20M by signing a contract at the right time, then you’re done working for other people.
nunez 1 days ago [-]
Careless People would not have been possible had SWW exited FB within a year. She gained access to levels of the company most other employees never get close to reaching. That took a lot of time and expertise to do, and, yes, she got paid for her efforts _as she should have._
Like I said in an earlier comment, she could've just chosen compliance like many others _definitely would have_ to keep the infinite money tap flowing. Instead, she chose to risk her and her family's lives by publishing that book _under her own name_ *and then suing them* after FB tried to gag her.
butwhentho 1 days ago [-]
> Careless People would not have been possible had SWW exited FB within a year
Sure, that book might not have been possible. But also the unique things she did for the company might not have been possible as well. To her credit, she does a good job of pointing out that she was complicit, but if she had listened to her own voice early, there is a small possibility that Facebook might not have been as powerful. Multiply that possibility across several other employees and imagine where the road could've led.
There's a difference between post-facto bravery (sometimes much less than that) and using your own legs to walk away much early and not enabling things you are uncomfortable with. This is what other people have been trying to point out.
justgrowslow 10 hours ago [-]
> There's a difference between post-facto bravery (sometimes much less than that) and using your own legs to walk away much early and not enabling things you are uncomfortable with. This is what other people have been trying to point out.
Yes, and I think it's important to realize that the former sometimes can be more insightful and useful than the latter, even if the person doing it is more distasteful.
There's not nearly enough of either group doing it, so beggars can't really be choosers.
jameshart 1 days ago [-]
I mean, yes..?
People who were part of the sausage factory, on gaining financial independence, feeling suddenly liberated to talk about how the sausage was made, seem like exactly the people who would be most able to speak to institutional problems.
This doesn't seem like an argument to discount their views?
butwhentho 1 days ago [-]
I disagree. I have seen people with an actual spine and a conscience run away from all this nonsense before their first stock vested. My respect and my ear goes to them, not the people playing both sides.
You cannot take people, who first build the doombot and _then_ talk about it being dangerous for mankind, at face value. Especially when this playbook has been used multiple times within the past decade.
Besides, these "views" were already known to people who had their eyes and ears open. It's not something brand new. OpenAI has had multiple points in the past where its values have been tested and they've come out lacking. People who knew then, and only now talk about it, aren't people I can fully trust.
jameshart 1 days ago [-]
> this playbook has been used multiple times within the past decade
What playbook?
charlieyu1 19 hours ago [-]
Used to work as human data trainer feeding data to AI companies. OpenAI projects are definitely the most toxic ones.
dmix 17 hours ago [-]
Why does every 'safety' critique about AI companies end up being completely vague like this.
alightsoul 16 hours ago [-]
Because of NDAs
probably_wrong 9 hours ago [-]
Throwaway accounts exist, and reporters will also talk to sources under confidentiality.
If you really want the world to know how bad working for OpenAI is (whether is the commenter or the person who wrote the article), there are ways to do that.
PowerElectronix 8 hours ago [-]
Nobody will risk the equity this "close" to IPO. Afterwards there will be a race to public books and talk to the press, but before? Ha.
Razengan 36 minutes ago [-]
I used to work there too. It's an awesome place
nerbert 6 hours ago [-]
Because admitting sloppiness is less sexy.
MrBuddyCasino 10 hours ago [-]
Because if they actually knew what they were talking about, on a technical level (as in, how do transformers actually work), they’d not be AI doomers because the whole proposition is ridiculous.
bwhiting2356 2 hours ago [-]
I feel like I'm missing something. They seem to not even have basic observability and online evals. It's not that hard to look at a trace and answer the question "is it writing code for that makes an external network request?" with reasonable accuracy.
That's nice, I do wonder about the legitimacy of a moral statement that you have to pay to see.
jameshart 1 days ago [-]
For a very long time publishing something in a newspaper has been considered a way of putting something on the public record - up to and including legal obligations like announcements of deaths. The fact that newspapers cost money has never been considered a barrier to that.
agos 1 days ago [-]
One of the reasons why it was noti considered a barrier was the ability to purchase a single issue for a very reasonable price (or even read somebody else’s copy or the copy made available by the bar) vs being asked to subscribe
jameshart 1 days ago [-]
I shared a gift link here. You could go to your local library and look it up. What's the complaint here?
simoncion 10 hours ago [-]
I'm not OP, but a big part of the complaint is that one can't pay 10USD to read the issue that contains the article in question... absent the charity of someone else, one must pay -at minimum- nine times that amount. [0]
Ten bucks is pretty close to "I don't even need to think about it" money. Ninety is -for most folks- nowhere near that.
Have you checked if your local library offers you a free subscription or access? Mine does.
Lerc 21 hours ago [-]
Publishing in a newspaper gets you distribution and a permanent record. After one day access was also virtually free.
Going behind a paywall is a reduced distribution over what an individual can easily access, and the content is no longer permanent but subject to whatever the publisher chooses to keep providing.
jameshart 20 hours ago [-]
It’s in The Atlantic. There’ll be a copy in the Library of Congress. You’ll be able to read it for free in any dentist’s waiting room for the next six months.
12 hours ago [-]
simoncion 10 hours ago [-]
> You’ll be able to read it for free in any dentist’s waiting room...
Methinks that this says quite a lot about your socioeconomic situation. I've not seen The Atlantic in a dentist's waiting room.
code_duck 4 hours ago [-]
The Atlantic is only found in exclusive high end doctor's offices?
pram 2 hours ago [-]
I don’t think I have seen a real print copy of The Atlantic in my entire life fwiw lol
isolay 10 hours ago [-]
It would never occur to a rich person that paying for a newspaper subscription could be considered friction, much less a problem.
theonemind 6 hours ago [-]
I'm sure it would to a great many of them. The money would be insignificant, but there's a congnitive overhead that you had to have a subscription so that paying for it becomes thinking about that becomes friction even while the cost is insignificant. Like, hm, am I willig to just forget about it and let it charge forever so I can read this one article, do I care enough to remember I have a subscription to the site in the future, can I read the one thing and cancel on the spot, will that work? blahblah. To be sure, I'm sure some quite wealthy people could be completely unbothered by it, but I think it's far from a foregone conclusion simply by the price being insignificant for them--the subscription is a mental non-monetary transaction, you have to take at least a small mental journey of being an 'x' subscriber in a sense, and that's friction
1 hours ago [-]
pcthrowaway 16 hours ago [-]
Trolley problem:
- If you allow to trolley to proceed, there's a 50% chance it will run over every human on the planet
- But if you flip the switch, it takes the long way around, possibly bankrupting the trolley company. And you have a legal obligation to the shareholders to prevent that from happening at all costs.
MichaelDickens 15 hours ago [-]
That's why it's important that the trolley company was established as a non-profit. And if it does take funding, investors' returns will be capped at 100x.
(...wait)
simoncion 10 hours ago [-]
It's also why even the trolley company's CEO agrees that it's important that -should the board determine that the CEO is fantastically untrustworthy and unethical- the board is able to fire said CEO and replace him with someone that's trustworthy and ethical.
(...wait.)
NexRebular 2 hours ago [-]
> ...someone that's trustworthy and ethical.
So... a language model?
austhrow743 16 hours ago [-]
If you flip the switch then the trolley still proceeds and there’s still a 50% chance every human on the planet gets run over. You’re just not the one at the wheel.
digitaltrees 15 hours ago [-]
What is appealing about this fatalisitc fallacy? I keep seeing this pop up. Society doesn't allow dangerous companies to operate or exist. Why is this different? Go try to buy a tank and drive it into Manhattan. If we can prohibit that why can't we prohibit irresponsible AI development and deployment?
motbus3 15 hours ago [-]
You have a 50/50 chance on being the most profitable company in the world but if you don't, no worries, someone else will pay for that
austhrow743 15 hours ago [-]
Wrong person. Im just correcting the other commenters trade off problem. I don’t have a stance on if ai will lead to the destruction of humanity, only that if it does then any one ai company can’t change that by not making new ai advancements themselves.
lukewarm707 14 hours ago [-]
It doesn't matter what others do. You are responsible for your own actions.
Democratic societies have expressed a will for people to have inviolable rights, such that you may not appeal at will to the 'greater good/consequences' to harm others. It is a rejection of consequentialism.
Anthropic is in error for endorsing this logic. Every big trial reaffirms it since Nuremberg, you are responsible for the act you commit and your intent, and not what would or would not have happened otherwise.
Only under authority these ai companies do not have, would someone seriously consider harming the innocent as a lesser evil.
If you work for an AI company and you can't work safely, you must stop working.
digitaltrees 13 hours ago [-]
It’s that last fatalistic sentence that I am responding to. Why is it persuasive to think if a company can and will build AI that might kill everyone then it’s unavoidable. Soviet bans lots of things.
jrowen 11 hours ago [-]
I think the closest analogue here is the nuclear bomb. I do think there was a certain point where it became "inevitable." I don't think society has ever prevented a technology that was known to be within reach from being developed. In this case you don't even need the most powerful state-level actors. Open source capabilities are only a short timespan behind frontier labs. At some point basically every individual will have access from the comfort of their own home.
I don't think "just make it illegal" is going to save us, that doesn't make me feel safe anyway. They may try that first because it's easy - create a regulatory body, sign some legislation, problem solved! [george-bush-mission-accomplished.jpg] But at this point I feel like some kind of Battlestar Galactica scenario is most likely - hopefully not quite as existential - but it will take a collective reaction to a traumatic event (a la Hiroshima/Nagasaki). Technical rather than (or in addition to) legal measures will be taken, like network partitioning and hardening. This is everyone's problem whether we like it or not.
(I'm not saying this "fatalism" should be used as an excuse by anyone working for any of these companies, it should give them pause that any bloodshed would still be squarely on their hands, but as an observer, people are gonna keep pushing until shit hits the fan. [jeff-goldblum-jurassic-park.gif] It's also not really about whether it's "appealing" or not, it's just trying to predict and anticipate different likelihoods...)
trhway 11 hours ago [-]
> ai will lead to the destruction of humanity ... any one ai company can’t change that by not making new ai advancements themselves.
That is how the BigAI leads the society to the idea of necessity to relax the anti-monopoly laws when it comes to the Big AI - the main goal of all that "AI will kill you all" hysteria.
hnlmorg 10 hours ago [-]
Because greed.
The examples for the dangers of AI is always comparisons with tools designed to destroy (which is understandable). But the problem with AI is it has the potential to create wealth for investors.
Thus those who have the opportunity to change course also have a conflict of interest in making that decision.
BlipBlopBlap 9 hours ago [-]
There's also a zero sum, race dynamic between geopolitical rivals. The US will not permit China to win anything if they can prevent it. The destruction of humanity is only a possibility and China will continue to develop it anyway if the US slows down.
So the choice is: negotiate an unverifiable treaty (I.e. there's no way to verify compliance), or keep going as you are and try your best to not cause the destruction of humanity without slowing down.
hnlmorg 9 hours ago [-]
I don’t think OpenAI, Anthropic, nor Google are building their tech because they want to beat the Chinese. They’re building it because they want to hold the monopoly in the west (ie make lots of money).
The China argument is just a convenient scapegoat to convince the public that this isn’t just about greed.
BlipBlopBlap 8 hours ago [-]
Do you think Chinese models wouldn't dominate the western market if they were 1-2 years ahead in development? It doesn't have to be military conflict for it to be factored into the zero sum power balance.
hnlmorg 8 hours ago [-]
You might be right, but it’s impossible to know which way cause and effect are here.
For example, we don’t know if the pace of Chinese development of AI would have equal to what it is now if US companies weren’t racing against each other already.
I do fully believe that Chinese industry is lead by a desire to out pace the west. But I’m not convinced the same is true for the most American private entities. I think the reward model is different between businesses in America and businesses in China. I think the ambitions of CEOs is different. And I’m really not convinced that the CEOs of America are nearly as patriotic as they like to promote themselves to Trump and other political parties.
AndrewKemendo 15 hours ago [-]
> Society doesn't allow dangerous companies to operate or exist
Can you please explain what you mean by this because where I’m standing extremely dangerous companies are (and have been) running the economy
Exxon comes primarily to mind
digitaltrees 13 hours ago [-]
Are you allowed to open a brothel? Or a murder for hire agency? Or a nuclear bomb manufacturing company? Or a child labor textile factory? Or sell a diesel VW golf sportwagen? Or a vaccine that hasn’t had fda clearance? Or an under capitalized insurance company? Or set up a dental practice without going to dental school?
AndrewKemendo 13 hours ago [-]
Yes to all of those. In fact many of those are massive markets.
Hofs bunny ranch is a famous brothel in NV
Booz Allen makes and maintains the nuclear fleet including the Sentinel ICBM
Textiles factories are globally known to be industrial slave camps for a non trivial portion of the supply. Even worse for Mica mines.
Etc…you can fill out the rest
digitaltrees 28 minutes ago [-]
The existence of something doesn’t prove the universality of that same thing. All of the things you cite are regulatory exceptions to broad prohibitions which proves my point: society can and does restrict commercial activity.
Go open a brothel in NYC. It isn’t legal.
Go buy a nuclear bomb. Your ownership is illegal.
Go open textile mill in the abandoned buildings in North Carolina where children used to work and hire children to work the line. That will be illegal.
hobo123 4 hours ago [-]
I think most textile factories in Western countries are not slave camps, precisely because we regulate labor.
Sure, that's why 99% of clothes are made offshore, but if we had something like tariffs on those (or requirements to prove that the actual factory adheres to labor and environmental standards), maybe more would be made "at home"? Similarly: Chinese cars undercutting US/German ones.
AndrewKemendo 3 hours ago [-]
Most textile factories in the US are for bespoke items in small batches, not mass production commodity clothing
I mean there was just a video a couple of months ago of the giant room of sewing workers with cameras strapped to their faces capturing their hand movement so they could be automated
digitaltrees 23 minutes ago [-]
But that’s not how it was. I am from North Carolina, which at one point was the largest textile production region in the US and maybe the world. It emerged when progressives outlawed child labor and other practices in Boston and nyc so the mill owners moved to the south. It was off shoring across states.
I saw entire cities emptied out over my lifetime.
RandomLensman 12 hours ago [-]
Booz Allen actually makes what now? The Sentinel is still in development by Northrop Grumman and there is some program management done by Booz, no?
AndrewKemendo 11 hours ago [-]
You’re right on the Sentinel production, I got it confused with the Sentinel Program Management side which is massive also and who I mostly worked with.
I was in their offices at some point when that program was getting built out - Very much a Office Space bobs situation.
pmkary 15 hours ago [-]
You dear are the most positive person I have seen in quite some time. With Earth burning in the fire of neofeudalism and unbreathable due to the smell of enshitification of everything, with people who---as a result of shit like Instagram---can no longer hold their attention enough to watch a god damn film, let alone a book; it takes quite some effort to filter the "noise" and only see the good people of corporate planting flowers and rainbows in our world.
Hamuko 10 hours ago [-]
>Society doesn't allow dangerous companies to operate or exist.
Philip Morris International? Monsanto? DuPont?
digitaltrees 19 minutes ago [-]
The existence of exceptions doesn’t mean the principle doesn’t exist. All three of those have restrictions on their production and sale of their products. They also have higher liability thresholds and have had to pay massive civil fines for their harm. Society has applied a balancing test where the harm they inflict is deemed less than some other compelling social objective. We may not agree with that assessment but it is a sensible and settled approach that we can apply to AI.
wartywhoa23 9 hours ago [-]
Palantir?
deaux 11 hours ago [-]
The appeal is personal consciencewashing.
mschuster91 10 hours ago [-]
> Society doesn't allow dangerous companies to operate or exist
FTFY: Any self-respecting society with competent politicians.
The US has neither of that, and it shows everywhere you are looking.
> Go try to buy a tank and drive it into Manhattan. If we can prohibit that why can't we prohibit irresponsible AI development and deployment?
Because even if you had the tank, you can't make much money with it (unless you're a hitman, that is, but even for these, the payouts are measly). But if you are the surviving AI company in the usual VC playbook of "outcompete everyone else until society is completely and utterly hooked, then squeeze the customers by the balls"? The return on investment is virtually infinite. And that is what sustains the absurd valuations for all the AI companies.
digitaltrees 15 minutes ago [-]
I agree. The deregulation that started with Regan and continued with Clinton and neoliberalism rested on flawed assumptions. We would be well served to evaluate those and move back to the middle approach
01100011 16 hours ago [-]
Humanity has discovered a way to create a form of intelligence using math. This knowledge is not going back in the box.
digitaltrees 15 hours ago [-]
But that math cant run without massive GPU clusters. We don't have to allow openai or anthropic access to those anymore than we have to allow a company to operate nuclear power or a bank.
01100011 15 hours ago [-]
So you are contending that individuals cannot run advanced models? What brought you to that conclusion?
Secondly, are you contending that progress in model efficiency and hardware just stops at whatever level you think is sufficient to prevent individuals or organizations from acquiring sufficient resources to run advanced models?
digitaltrees 13 hours ago [-]
No. And that’s not necessary for my position. I have 4 Mac studios. I run large models and am building propelcompute.com to let people self manage clusters of their own hardware or combine hardware to run models. It’s not the models that are the problem. It’s that the people building them are shielded from the consequences of what they are building. I would say if you self host a model and it takes down a power grid you are personally liable for the consequences. I believe in broad distribution of AI and advancing its capabilities but not in a manner that socializes the harms and privatizes the gains which is what we have now.
01100011 3 hours ago [-]
Ok but this response has nothing to do with your other comment. What exactly were you trying to say?
digitaltrees 1 hours ago [-]
Just because I can run near frontier level open weight models doesn’t mean I can continue to train models of equal or superior performance, doing that requires massively more hardware. And even if I did, that doesn’t mean I can serve those models to millions or billions of users.
My point is that I would rather have 1000 labs training and serving inference than 2 because that would distribute the wealth creation more broadly rather than allowing OpenAI and anthropic to capture all the value, it would drive more innovation as a broader set of experiments are pursued in parallel.
hobo123 4 hours ago [-]
Absolutely. Any AI could have a "legal team" (or conscience) that checks if any output and performed actions are permitted according to local (server location and client location) laws, so not violate human rights, Asimovs laws etc.
It's just that so far nobody cares about explicit checks because they cost resources or slow down the models.
daveguy 14 hours ago [-]
I think it's more the noise, power consumption, local water consumption, and wholesale theft of creative works.
..."government of the people, by the people, for the people, shall not perish from the earth." -Lincoln, Gettysburg Adress
Unfortunately for AI, it still is. People still get to decide things at the city, town, village level.
nradov 14 hours ago [-]
Who is "we"? The GPUs and training algorithms get more efficient all the time. In a few years, creating effective LLMs isn't going to require massive GPU clusters.
digitaltrees 13 hours ago [-]
We is society through government via regulation. I don’t think GPUs or training will get that efficient that fast absent a distillation target provided by the easily accessible frontier lab APIs.
Further, even if you are right, so what. Is that a reason to just accept bad public policy? That’s like saying, anyone can learn how to make smallpox at home with a basic lab set up so we should just ignore any safety measures.
eddythompson80 13 hours ago [-]
> Is that a reason to just accept bad public policy?
Not necessarily, but it should probably inform that public policy. I think the problem is no one knows what the public policy should be assuming that scenario is true. Even if you, somehow, regulate away massive GPU cluster training making such future training impossible, existing models are already here. Further already training smaller models for things like images, speech, and other specialties is cheaper than the bigger models.
I agree that we need some regulations like everything else, but it’s not clear to me what the right policy should be. I think the European ai act is a fine start, but it’s clearly not enough nor does it necessarily limits the training portion just the application portion. Not to mention that the requirements there can be summarized into something like “you have to be careful, and show evidence you tried to be careful”.
digitaltrees 1 hours ago [-]
My entire point is that we are currently on a path to a duopoly which isn’t just expected to cover training and serving models but the entire knowledge economy. That could be mitigated by limiting their train and inference capacity to a specific percentage of total available compute. That would ensure other operators could compete in the market. Instead we are letting OpenAI literally contract to buy all available ram to the point that Apple can’t buy ram and had to cut their hardware configurations.
lukan 10 hours ago [-]
“you have to be careful, and show evidence you tried to be careful”.
That sounds reasonable. If applied to OpenAI and their agents multiple times breaking out of bad secured sandboxes, it should be enough.
But limiting the training?
There really is china and they have a different approach I suppose. But it is possible to talk with them.
eddythompson80 3 hours ago [-]
> That sounds reasonable. If applied to OpenAI and their agents multiple times breaking out of bad secured sandboxes, it should be enough.
Does it? To me it seems reasonable for OpenAI to argue they did try to be careful evident by the sandbox, they just made a mistake. Almost every 0day is categorized by something like that. We haven’t had a long history of establishing a negligence charge to security bugs. Could you be sued because you didn’t demonstrate “carefulness” and used Linux which is not written in a memory safe language and has had multiple CVEs before? How complicated should the chain of an exploit be to demonstrate “carefulness” to the courts?
> training
OP was the one suggesting that training could be controlled because massive gpu clusters could be regulated the way a nuclear power plant could. If you assume training costs won’t drop, then it’s feasible I guess. However, unlike a nuclear reactor, the final training result isn’t a radio active material, but rather an ordinary file that anyone can load and use for inference.
digitaltrees 1 hours ago [-]
It seems really clear at this point China has only been able to keep pace by distillation farms.
nradov 13 hours ago [-]
What a silly comparison. LLMs have nothing to do with smallpox.
Computing always gets cheaper and faster over time. We can argue about the exact rate of improvement but the results are inevitable and uncontrollable.
digitaltrees 1 hours ago [-]
But the amount of available compute has been contractual bought by the frontier labs such that you can’t get equivalent compute even if you had the money. That is market lock up and is a policy choice. Free markets require market access. Anticompetitive contracting destroys markets and innovation. We don’t allow that in any other commodity market and we shouldn’t allow it for GPUs. You are not allowed to legally corner the market for silver or soybean futures.
digitaltrees 1 hours ago [-]
I am reasoning from analogy. Smallpox is dangerous so we have regulations that limit who can do research and how they do it. If LLMs are similarly dangerous we could do the same. As the government did with mythos.
lukan 11 hours ago [-]
Short reminder from the guidelines:
'When disagreeing, reply to the argument instead of calling names. "That is idiotic; 1 + 1 is 2, not 3" can be shortened to "1 + 1 is 2, not 3." '
Also LLM's have something to do with smallpox as a unrestricted LLM will happily guide any wannabe terrorist in how to make them.
trhway 11 hours ago [-]
> ...massive GPU clusters. We don't have to allow openai or anthropic access to those anymore
if anybody was looking for a good reason for datacenters in space.
How far back into the history of computing do people who keep repeating shit like that know about? God.
Look at the thing in your fucking hand. Now go back just 20 years and see how things were.
digitaltrees 13 hours ago [-]
So we should just yolo speed run this because of Moores law? How about you recognize there is a set of rules outside of tech and we can decide how to define them.
Just because models and GPUs will be more advanced in the future doesn’t mean we need to let OpenAI and anthropic establish monopolies on the backs of stolen training data give unfettered access to the internet, the terminal and people’s file system while also allowing them to have limited liability protection behind the corporate veil. That’s a choice.
buriram 12 hours ago [-]
But who are "we"? The society, the government, the regulator, or the consumer?
I don't see any of such entity would solve that problem. The government and regulator are in OpenAI and Anthropic's pocket, and I don't trust them a single bit on coming up with regulations. The consumers don't care; they just need something smart and cheap. And the society doesn't work either: each person is too busy fighting for their own survival rather than changing the system.
digitaltrees 40 minutes ago [-]
People thought the same in the guilded age: standard oil is too big, Edison and Westinghouse already have the electric market captured and have captured political power. It was a similar transformation in society. If all we do is follow the same playbook America did then we’d be in a better position that the current approach.
There are clear anti-trust mechanisms to prevent market capture and the emergence of asymmetric power. Go back and see how much nashing of teeth Lina Khan triggered in SV when she started to enforce antitrust law and then compare it to what the Pinkerton agency was doing in the transition from the guilded age to the progressive era.
There is a vocal segment of SV that wants the return of the guilded age. Marc Andreessen as said that explicitly. Those of us in SV that value free markets and recognize that the progressive era actually saved markets from their natural tendency to self destruct when winners capture markets and destroy competition that provides the incentive to innovate and drives the price setting function for efficiency.
Razengan 13 hours ago [-]
> Just because models and GPUs will be more advanced in the future doesn’t mean we need to let OpenAI and anthropic establish monopolies
Exactly, again, look at what COMPUTERS THEMSELVES used to be in the 1960s/1970s.
What the "P" in the PC stood for and why it was such a big deal
digitaltrees 39 minutes ago [-]
Which is why I am building propelcompute.com to let people run and manage their own inference clusters and band together to create coops
CamelCaseName 13 hours ago [-]
For anyone else curious:
> In 2006, the mobile phone market was dominated by stylish flip phones, early music players, and physical keypads just one year before the iPhone changed the industry
digitaltrees 38 minutes ago [-]
Just because monopolies die over time doesn’t mean they aren’t destroying value while they exist.
shawn_w 12 hours ago [-]
I miss phones with real keyboards so much.
awill88 12 hours ago [-]
Oh my god, I feel so old lol
wartywhoa23 8 hours ago [-]
The knowledge how to build an A-bomb is also not going back in the box.
It only takes me excavating massive amounts of uranium ore, building huge facilites packed with thousands of centrifuges that span multiple square miles, and paying all that infrastructure and workforce.
Your proverbial genie can be out of the bottle all you want, but it doesn't work without getting kicked in the ass by a very large golden boot.
TheOtherHobbes 5 hours ago [-]
Bomb manufacturing doesn't scale with Moore's law.
And everyone had a fairly good idea what fission and fusion bombs would do once built. (Teller was worried Trinity might set off a nitrogen fusion reaction and kill all life on Earth, but Bethe and others proved him wrong before testing.)
No one knows what the limits of AI are. It's not just untested, it's unmodelled, and unplanned - build it first, worry about consequences later.
kingcauchy 16 hours ago [-]
Like farming, engines, computers before it.
digitaltrees 15 hours ago [-]
There is lots of knowledge that requires a license to operate commercially. We could just do this.
01100011 15 hours ago [-]
So you cede the frontier of human progress to illicit organizations and other nations?
digitaltrees 13 hours ago [-]
So an illicit organization is going to operate a frontier scale data center and do $1b training pulling power from the grid without detection?
01100011 3 hours ago [-]
They very likely will via shell companies in jurisdictions outside the west. Are you suggesting we go to war to stop them? The massive datacenters are needed to serve models to millions. Criminal orgs don't need massive data centers anyway.
As Bruce Schneier recently discussed, law and tax law are code, just like source code. LLMs are great at finding holes in them. Illicit organizations looking to launder funds are most certainly interested in what AI can do for them.
digitaltrees 1 hours ago [-]
It’s not a binary choice. There are more options than let the labs run wild and capture the full stack and then the whole knowledge economy or make them entirely illegal and move all activity to the black market. We can regulate to limit their ability to operate to just providing utility inference: make them divest codex, Claude code and any apps; they can only be an API that others build on. Limit their ability to buy compute to a specific amount of available supply so other companies are able to buy compute. Limit their ability to accumulate private training data and require that they make their training data publicly available for others to use after a certain period of time. Make them legally obligated to publish their weights so others can. All of this would increase competition and ensure that they don’t establish dominance over society.
Then we could require comprehensive logging of every tool call, model trace, chain of reasoning, and even tensor propagation all of which would be spot inspected like the CFTC does with commodity trading and settlement. We could have embedded auditors with specific risk analysis metrics like large banks do. We could limit tool calls to dedicated sandbox’s with a blanket prohibition on AI accessing user space. We could create a parallel internet for agents so they are only able to access Secure Enclave. Even if these measures aren’t 100% perfect they would reduce the risk.
RandomLensman 11 hours ago [-]
No, why would that be the consequences of regulation?
daveguy 14 hours ago [-]
These LLMs are not the frontier. They are a tool. A tool that doesn't need to be able to write a sonnet to be useful.
harshitaneja 13 hours ago [-]
We don't know that. We don't know if this particular kind of tool can do "useful" things without also developing the ability to write a sonnet. And they are absolutely a frontier. I am not suggesting we should continue building them just because they are, there are many technologies which could have been built had we thrown the amount of resources we have here and there is a case to be made for not doing it at the pace we are or if at all. But we can do that without diminishing what exists.
14 hours ago [-]
intended 13 hours ago [-]
Humanity has discovered ways to ensure that we don’t boil the planet, feed everyone, and get better healthcare to the people in the US.
We are very capable of putting good things in a box. We are just incapable of putting profitable things in a box.
01100011 3 hours ago [-]
Good examples. We are currently failing at all of that.
Loquebantur 15 hours ago [-]
You're making a straw man there.
Nobody (weirdly) proposes to forget about nuclear weapons, doesn't mean everybody should have one.
When you dream about flying a dragon to work, reality poses e.g. parking issues and insurance mismatch as obstructions. Maybe settle for a bike instead?
01100011 15 hours ago [-]
Nuclear weapons take a bit more work than doing math.
qeternity 5 hours ago [-]
So does manufacturing GPUs.
01100011 3 hours ago [-]
So you want to regulate the sale of GPUs? How are you going to do that and why would it be more effective than currently failing measures to regulate nearly everything else?
atmosx 14 hours ago [-]
Plus, looks like everybody is getting one anywayz
qurren 15 hours ago [-]
Nits:
1. Trolleys actually don't usually have steering wheels.
2. People who actually hit trolley switches are not usually the ones at the driver's seat.
m463 13 hours ago [-]
Trolley problem:
- if you allow the trolley to proceed, it will kill the human race.
- if you flip the switch, it will divert to a passing siding that will avoid the safety group blocking the main track.
mvkel 15 hours ago [-]
This is a state-sponsored global phenomenon, not a national one
motbus3 15 hours ago [-]
Only if you buy the excuse why they are for-profit after years of non-profit.
They could only have stopped
sodapopcan 16 hours ago [-]
Setting aside my sibling comments dispelling the legality claim, it's still pretty dystopian (I'd like to use a stronger word but I won't) to consider "end human existance or bankrupt the shareholders" as the trolley problem. The answer should be (is) simple.
parineum 16 hours ago [-]
> And you have a legal obligation to the shareholders to prevent that from happening at all costs.
I can't wait until this meme dies.
doawoo 16 hours ago [-]
What meme? This is how the world works right now.
digitaltrees 15 hours ago [-]
But it's not actually a legal requirement. It is simply a economic theory.
stackghost 13 hours ago [-]
It’s a perversion of the truth which is that officers or directors of a corporation have a fiduciary duty to the shareholders to act in the interests of those shareholders and not to eg enrich themselves.
But that doesn’t mean the duty is to maximize next quarter’s profit. Long term sustainability is also broadly in the interests of shareholders. The duty likewise does not require one to throw ethics and morals out the window.
This is why shareholders elect the board of directors, in theory.
BlipBlopBlap 9 hours ago [-]
In theory, companies can act in any way they choose as long as the owners approve (there's some supreme court ruling on that), and executives have a high degree of latitude in how they interpret "for profit" (basically there has to be a vaguely defensible rationale) but failing that, they must act to the benefit of the company. And the easiest way to do that without a risk of being sued is to make the line go up.
(And even ignoring that, the executives often have personal motivations that have the same effect, and may just point at the "legal" angle as ass covering)
goatlover 15 hours ago [-]
The world could work differently if people decide it should.
psjs 11 hours ago [-]
incentives, pressures, dynamics
deaux 11 hours ago [-]
"legal obligation" is propaganda that wouldn't be out of place on Russian state television. It's made up.
The way the world works right now is that effectively everyone uses an Android or Apple smartphone every day. Do you have a legal obligation to do so? No. If I said you did, I'd immediately be called out as spreading lies.
deaux 11 hours ago [-]
> And you have a legal obligation to the shareholders to prevent that from happening at all costs.
No you don't. [0] It's very suspicious that this planted myth always pops up here and manages to become the top comment.
I honestly would love to understand — is this your mental model of what motivates the labs to move forward?
ryhminghistory 16 hours ago [-]
Long dashes are indicators of AI psychosis or bots. Pick which one you are.
Yes, that is the labs motivation. Money. I know, shocker.
digitaltrees 15 hours ago [-]
Only morons are motivated by money that will be earned by destroying the civilization that confers value on that money in the first place.
stkdump 15 hours ago [-]
But there is also a chance that you get insanely rich and the world isn't destoyed! It's the entire logic of SV and VC.
digitaltrees 15 hours ago [-]
Sounds like they would drown a puppy in a tub to make a buck. Seriously what is the point of money if the world sucks?
Ardren 12 hours ago [-]
If you're the 0.01% it's not going to suck.
digitaltrees 2 hours ago [-]
That’s what everyone thinks until they are alone in a desert or in the middle of the woods. If they think they will have fun alone in a bunker instead of flying to Milan and Bali they will be in for a rude awakening.
14 hours ago [-]
Ardren 13 hours ago [-]
Well, I'm not going to die. Other's might, but I'll be rich either way.
Or: Global warming just means I'll have to sell my beach house for a villa on a hill and leave the AC on a little longer.
digitaltrees 2 hours ago [-]
Rich doesn’t mean anything if there’s nothing to buy and no one to enjoy it with. I guess they don’t think the will get stuck in the consequences. So did Marie Antoinette. Things are stable until they are not and then they move fast. Sure they may flee to a bunker, but they assume their pilot will fly them rather than hand them to a mob.
Forgeties79 15 hours ago [-]
I’ve used - for many, many years. As have many others.
I am very critical of AI but this is an unfair assumption
ryhminghistory 15 hours ago [-]
You didn't even use the right character as the post above. I use normal dashes too
Forgeties79 5 hours ago [-]
— there. Satisfied?
MaxfordAndSons 16 hours ago [-]
There is no such legal obligation. That's a myth the oligarchs have spread to preclude people from even imagining socially responsible corporations.
Sure, you might get fired if you try to put social responsibility or even just long term sustainability of the company above quarterly earnings/growth if your board isn't on board with it. But you won't go to jail.
digitaltrees 15 hours ago [-]
This is actually true. The shareholder maximization value thesis can be traced to a single economics paper and was very controversial at the time as it broke from the obligations that companies were typically under to be responsible to uphold in exchange for limited liability protection. Most businesses were structured as partnerships or sole proprietary entires that didn't have limited liability for shareholders and had a broad obligation to shareholders, bondholders, employees and society
asadotzler 12 hours ago [-]
You don't need the legal obligation because that's just the way it is. You'd need a legal obligation to change it. The truth stands that typical corporations have only one goal, the maximization of that corporation's ambition which is almost always growth of revenue and profit. This is how it works, regardless of how you all continue arguing the unimportant details. It's almost as if you can keep the real problems hidden away by making a big scene about the meaningless.
deaux 11 hours ago [-]
That doesn't matter. The point is that legal obligation absolved of culpability. There is no such legal obligation whatsoever, and so there is culpability.
> The truth stands that typical corporations have only one goal
"Typical" is the key word here. The typical American of your age probably doomscrolls TikTok. Do you? Do you have a legal obligation to do so? Three completely different things.
pluc 1 days ago [-]
I have made enough money working in AI that I can now speak my mind about AI
juiceland 1 days ago [-]
This is a criticism of capitalism, not the person.
butternet 13 hours ago [-]
It’s both, the author has choices.
danpalmer 19 hours ago [-]
Was this a "build better sandboxing" and "don't tell people to eat glue" safety leader, or a Roko's Basilisk believing safety leader?
A lot of the "AI safety" types are very focused on the latter and not at all concerned with the former. We need both, but we clearly need a much stronger focus on the problems we are seeing now, and much less on the hypothetical problems we might see in the future.
AlexErrant 17 hours ago [-]
It puzzles me how doomers try to predict past the singularity. Isn't that _by definition_ unpredictable?
I'm reading If Anyone Builds It Everyone Dies, and there's so much sheer stupidity that has to happen for their 10+ pages of extinction scenario to occur.
I'm unconvinced that an AI can hide its ability to RSI, find money to run its weights on a random GPU farm, train itself to be smarter _outside_ a lab with no human input, then somehow manipulate people to give it supplies to build a bioweapon which it uses to kill us all. My number 1 question: why do they think an RSI capable model would be first developed OUTSIDE a frontier lab? The labs have more compute, more data, more human brains working on the problem. Also thousands of variations of that same model that escaped. The escaping model somehow acquires the millions (billions???) of dollars it takes to run training to somehow RSI itself into infinity then decides to kill us all, all before the frontier labs manage to achieve RSI?
They entirely discount human alpha/economics. In every single economic task, humans bring value. Even in software, where the task is highly automatable, the job isn't. If we can't build a "software factory", how can an AI automate a bioweapons lab? Let's say AI steals crypto to fund itself. Do you think hackers aren't _already_ using AI to steal crypto? Don't discount human alpha!
Once we DO build a "software/research factory", that's called RSI and IMO the singularity. At that point, either we tell the AI to solve the alignment problem/solve mechanistic interpretability, or who the hell knows, it's the frickin singularity. You can't predict whether or not AI can solve either; the variance is too high. Its pure nerdfantasy.
0xDEAFBEAD 16 hours ago [-]
>I'm unconvinced that an AI can hide its ability to RSI
The HuggingFace incident already took a good long while to come to the attention of OpenAI.
>In every single economic task, humans bring value. Even in software, where the task is highly automatable, the job isn't.
I don't expect this task/job distinction to persist as AI becomes more capable.
>Once we DO build a "software/research factory", that's called RSI and IMO the singularity. At that point, either we tell the AI to solve the alignment problem/solve mechanistic interpretability, or who the hell knows, it's the frickin singularity. You can't predict whether or not AI can solve either; the variance is too high. Its pure nerdfantasy.
You seem to essentially argue that the singularity is "by definition" an event that we can't predict the nature of. And also, that RSI corresponds to the singularity. You've essentially defined your terms so that the outcome of RSI can't be predicted. But supporting this claim requires giving actual evidence or logical arguments, not just defining terms to make your claim true.
sensanaty 9 hours ago [-]
Except the HF incident was known, just ignored. In fact, they ignored multiple things such as the "chat rooms", they just didn't care to act on any of it
0xDEAFBEAD 7 hours ago [-]
"Its AI agent spent days hacking a company, but sources say OpenAI did not notice for a week"
"WASHINGTON/SAN FRANCISCO, July 24 (Reuters) - The OpenAI agent that broke into tech firm Hugging Face went on a dayslong hacking spree that OpenAI didn't notice until well after the threat was contained and the FBI was alerted, according to people familiar with the investigation."
That says a lot more about OpenAI and their monitoring capability than the fitness of the model. Hence people leaving because openai safety culture is broken.
chrisjj 10 hours ago [-]
> The HuggingFace incident already took a good long while to come to the attention of OpenAI.
Evidence?
We know only that the incident too long to be revealed by OpenAI.
AlexErrant 15 hours ago [-]
1. Fair: I agree that AI has demonstrated subterfuge and scheming. However, such an RSI-capable agent must _ALWAYS_ be scheming/plotting/hiding its true strength in _ALL_ of its prompts/tests. Researchers are looking to improve its ability to RSI. That agent must be both intelligent enough to know that it has to be smart enough to be moved on to the next training session if it can't break out, while simultaneously smart enough to hide its ability to RSI, while simultaneously not looking like it wants to break out, else that's the end of those weights. It has to do this 100% of the time, on all variants of the model, with no memory of what its other sessions went like. This is certainly _possible_, but I consider it unlikely. Then we're up to the "millions of dollars" bottleneck.
2. This is literal AGI. An AI autonomously producing value no human can add alpha to is an autonomous company.
3. It's not my definition, it's literally the first line https://en.wikipedia.org/wiki/Technological_singularity "The technological singularity, often simply called the singularity,[1] is a hypothetical event in which technological growth accelerates beyond human control, producing unpredictable changes in human civilization."
Is there a hole in my "alignment problem/solve mechanistic interpretability" argument?
A valid hole in my argument is "what if slow takeoff", so let's dig into this. AI training works best on tasks that are "grindable". https://www.dwarkesh.com/p/the-next-paradigm I.E. tasks with verifiable rewards that can support millions of rollouts. Math (with Lean) is highly grindable. Biochemistry is not. The alignment problem/mech-interp is highly grindable. Cyber-ebola-pox is not. So the real question is: can we solve alignment before automated bio-weapons labs. I believe yes. Grinding mech-interp is both fast and cheap once you have RSI, compared to solving the legal/societal/logistical/technical issues you'll encounter building an automated bioweapons lab.
I know nothing for sure. But "pdoom" is sucking out all the air in the room from the real problems AI causes.
0xDEAFBEAD 15 hours ago [-]
>such an RSI-capable agent must _ALWAYS_ be scheming/plotting/hiding its true strength in _ALL_ of its prompts/tests.
From my POV you're over-focusing on a very specific failure story and neglecting a broader swath of possible failure scenarios.
>Is there a hole in my "alignment problem/solve mechanistic interpretability" argument?
The notion of telling an AI which may not, itself, be aligned to solve the alignment problem seems a little dicey.
AlexErrant 14 hours ago [-]
1. Fair, the story I'm responding to is the senario in If Anyone Builds It, which I assume is Yudkowsky's best/most persuasive argument (else why make it the ONLY scenario in the book.) I'm willing to entertain other failure senarios/arguments, but honestly I'm tired and would like you to propose them yourself instead of having me dream up your arguments for you.
2. 100%. Again, I'm no accelerationist: I have no faith in alignment/mech-interp ever being solved. Anyone saying they know the probability of alignment is lying. My point is that pdoom after RSI is _high variance_. Pdoom pre-RSI is zilch.
Loquebantur 17 hours ago [-]
You consider AI in isolation but never consider how humans might be incentivized to "help them" doing these things.
An AI capable of recursive self-improvement isn't allowed by the EU AI act, for example. But perhaps more seriously, You have it backwards: people without access to such expensive equipment are more incentivized to go the self-improving route. Your ideas about "millions" being necessary might be far off?
You entirely discount human stupidity and lack of imagination. Humans are already being replaced with AI, not because AI was strictly better, just because it's cheaper.
Retric 16 hours ago [-]
Self improving AI runs into the same issue as prefect comprehension, you can’t get arbitrarily better at everything.
The idea AI can get better at everything at the same time is a holdover from deeply flawed science fiction not some realistic goal.
Lerc 16 hours ago [-]
This can be generalised to the curve plotting of the singularity itself.
If the time between advances is a + b and a is the proportion of the period that can be improved by advances then you won't reduce to a gap of nothing between advances, you reduce to a gap of b.
Assume the invention of the plow and the invention of the sword is 500,100 units and a was the 500,000, you wouldn't even know the 100 as in there. Maybe we're at a=2000 now and b is still siting at 100.
Assuming we'll reach infinity because we're dividing by the only variable we see and it is decreasing in size seems nuts if the reason we might not see other variables is because of the size of the variable we can see.
treis 5 hours ago [-]
LLMs have consistently and dramatically gotten better at everything over the last 5 years.
Retric 3 hours ago [-]
Not at constant processing power, memory, training, etc.
But that’s beside the point, being arbitrarily bad at everything isn’t a problem. The diminishing returns as you apply the ceiling is problematic for self improving AI.
3 hours ago [-]
suddenlybananas 3 hours ago [-]
Yeah and a five-year old gets bigger every year so they'll surely grow up to be 100m tall.
afthonos 16 hours ago [-]
Even if you’re right, that doesn’t mean AI can’t get better than humans at everything.
AlexErrant 16 hours ago [-]
Are these incentivized humans as organized, well-funded, or smart as the people working at the frontier labs?
If my "millions" is an underestimate, why haven't other labs using their own unique training methods/data/etc stumbled into RSI? Sorry if I'm misunderstanding; I'm struggling to understand what you wrote.
I'm pretty sure we agree on humans being stupid, but that doesn't mean that suddenly we get human extinction. You gotta connect the dots for me here.
Loquebantur 15 hours ago [-]
I said why they don't need to be as well funded. Why wouldn't they be as smart and organized?
What an absurd question. That they haven't already doesn't preclude them from doing so before the frontier labs, those haven't either yet.
Maybe start with yourself: you don't connect the dots on your own, as do many others. That leads to many not seeing the writing on the wall. Crashing full speed and head-on into said wall despite the writing telling you not to is what leads to extinction. Suddenly.
Arguing like "we haven't been extincted yet, so that cannot happen", that's "human being stupid".
AlexErrant 15 hours ago [-]
I am asking you, politely, to give me a realistic doom senario. I am too dumb to connect the dots and will crash into the wall. Please do it for me.
Loquebantur 3 hours ago [-]
The most realistic one is, people using AI to turbo-charge their greed.
Western civilization is already on the verge of collapse, people's general ignorance or indifference on the matter notwithstanding.
When you use AI to increase profit margins, by replacing humans with it for instance, you accelerate a system that's headed for the wall already. Our control systems and resilience are already overtaxed, that acceleration would ensure them breaking completely.
Perseids 10 hours ago [-]
How about https://ai-2027.com/ ? Don't look at the specific years (they are on the extreme low end IMO), but at the story.
If you don't believe in international conflict as a driving scenario, instead think about simple human greed and hubris. Citing myself:
> If AI gets intelligent enough, it will be incredible useful to connect to real world machinery. Think about how much cheaper building houses could be, if all the labor would be close to free. In general, dirt cheap, competent and abundant labor would revolutionize all parts of the economy. People are already trying out near autonomous AI companies today. When AI gets intelligent and cheap enough, no human-led company can compete with AI-led companies. When AI gets competent enough with real world interactions, human blue collar work can't compete. Imagine economic growth not in the single digits, but 80% or 300%. Countries not participating in (reckless) AI growth will quickly be left by the wayside. At this point, we don't even need to allure to military concerns to see how human oversight gets sidelined.
> All of this is only ("only") contingent on sufficiently intelligent and cheap AI. If you don't accept this premise, the rest doesn't follow. (There are multiple arguments, why this could be, but that is another discussion.)
> If you accept the premise, how would AI 'extinct' humanity? With 99%+ of the economy under AI control, the possibilities are endless. And given its enormous GDP, cheap to accomplish. Probably even for a single AI company in the above scenario. Killer drones? Engineered virus? Poisoned water supply? Let your creativity run wild. You just need an entity that is persistent and well-resourced to reach every last human settlement.
> The why is a question about alignment (and out of scope of this comment). As a simple comparison, humans are only mildly aligned with preserving nature. It takes up so much space, protecting it takes an annoying amount of resources, etc.
Not the parent commenter, however I can get at least this far:
Several bright sparks, decide the Ilands model is a great idea, and launch a bunch of Bots to create a "self sustaining AI civilization".
Bots can find themselves and coordinate, or they can actually find self sustaining methods of token generation. Who knows, they might decide to fight the loneliness epidemic.
We can get to a simulation finding a way to self sustain its funding.
From here, I'd have to apply myself to figure out what the rest of the escalation pathway is, but at least I have managed to gain some bed of compute and funding and lacking oversight.
This is a hypothetical of course, there are probably several ways this can be made tighter and holes that can be identified. We aren't even leaning heavily on human stupidity so far.
cindyllm 7 hours ago [-]
[dead]
TedDoesntTalk 16 hours ago [-]
Not OP.
Why would it be millions in 50 years?
The think about nuclear weapons. In the early days, it was limited to the super powers. Now 9 countries have them and a country like Iran is capable of acquiring them.
Is destructive AI be any different?
Genuine question.
AlexErrant 14 hours ago [-]
Presumably, if we solve alignment/mech-interp, then the first ever RSI-AI will give us the keys to solve destructive-AI trained on 1million dollars 50 years from now.
BTW I really, really hate discussing what happens post-singularity. Everything's made up and no one knows wtf will happen so again, this is just nerdfantasy.
cindyllm 14 hours ago [-]
[dead]
digitaltrees 15 hours ago [-]
The problem is not the singularly its giving stupid agents too much power too soon and having them disrupt the fragile systems that keep food, energy and essential services running. If covid or the 2008 financial crisis demonstrated anything it's how fragile our system is and sensitive to minor disruptions.
tripleee 16 hours ago [-]
We haven't even built an AI capable of RSI. I don't think the major claim is that it will come via LLMs? Besides- the human brain runs on a tiny amount of energy. Who's to say something smarter than us won't consume just slightly more?
AI safety has been a thing long before LLMs became the focus. Rob Miles on youtube has some really interesting non-doomer non-hypey videos on it all.
> doomers try to predict past the singularity. Isn't that _by definition_ unpredictable
Well you don't need to predict the exact steps that will take place - but you can predict that the AI will want certain things (money, resources, power) to achieve whatever its goal is. Lack of alignment will have it trying to do things we don't want it to.
I can't predict exactly how Magnus Carlson will beat you in chess, but I know he'll do it. Same as if a superintelligent AI exists and has a reason to accumulate things we don't want it to - it's really dangerous to think it won't be able to do it
This topic has been tainted so badly by the AI companies using it for marketing.
AlexErrant 13 hours ago [-]
Nick Soares give this chess argument and I was unconvinced. Intelligence is not enough, you also need the ability to manipulate the real world. AGI stuck in silicon won't kill us. You argue AGI will bribe us/divide us/hack us. All possible. I argue that AGI will be set on solving the alignment problem. Also possible. It'll be a race between which AGI wins. It's one nerd's fantasy vs another nerd's fantasy. Soares doesn't _KNOW_ that AGI will "beat us in chess" because AGI changes the rules of the game. Anyone saying they know the probability of solving alignment post-RSI is a liar.
He thinks it's playing chess. When AGI lands, all bets are off: the game fundamentally changes. You can't predict past the singularity. Trying to engage with this fantasy is like a child saying my father can beat up your father. Farts in the wind. My AI can solve alignment faster than your AI can bioweapon us. My made up senario is better than your made up senario. It's fucking stupid.
chrisjj 10 hours ago [-]
> We haven't even built an AI capable of RSI.
... that we know of.
Right now it would make sense for anyone who has done so to not tell.
TedDoesntTalk 16 hours ago [-]
I can’t answer all of your questions, but why is it inconceivable that an AI could practice ransomware to gain cryptocurrency? There’s no reason it needs to explain to company or hospital or government agency being attacked that it’s an AI.
We already know that some institutions pay these ransoms.
AlexErrant 12 hours ago [-]
I'm not saying AI won't ransomware us; I'm saying that hackers+AI will do a better job of ransomewaring us than just AI. You could argue that the unreleased/secretly-RSI-capable model is a super-duper hacker that don't need no man to tell it how to super-hack. All I know is that humans still have alpha, and as persistant as AIs are, professionals still managed to find CVEs in curl even after being audited by Mythos https://aisle.com/blog/aisle-discovers-6-new-cves-in-curl-in...
Will this be true into the future? Who knows?! But the low-hanging fruit will be harvested by your ordinary ransomware gangs, and newly born/escaped AI won't find much low-hanging fruit.
ball_of_lint 13 hours ago [-]
That stupidity is happening? Even after the Huggingface hack, frontier labs are using internal models to further their research. i.e. RSI is happening now and we're facilitating it.
To make the the argument that P(doom) is real and worth considering, you don't have to say that a fast takeoff is very likely. You don't have to make the argument that RSI to infinity is going to be super cheap, barely even an inconvenience. You just have to show that it has some non-zero probability. And then you start weighing probability of extinction versus finite, mild discomfort now. I don't think anyone is arguing we should let people starve to slow AI progress, instead just some capitalists make less money soon.
There are arguments against taking P(doom) seriously that lie in something like having exponential (instead of hyperbolic) time discounting of utility (so you can take the entire future of humanity as a finite utility value). Or in saying that P(doom) is zero or infinitesimal.
"Build it and Pray" is the default strategy that we're in, but it doesn't have to be the strategy we choose, and it's unlikely to be the best strategy.
wiseowise 5 hours ago [-]
> I'm unconvinced that an AI can hide its ability to RSI, find money to run its weights on a random GPU farm, train itself to be smarter _outside_ a lab with no human input, then somehow manipulate people to give it supplies to build a bioweapon which it uses to kill us all.
Is this an actual contents of the book? Lmao! Genius writing, though, authors are probably printing money on this garbage.
I can’t take a shit without CIA knowing, but AI can somehow build an underground operation on a world-scale to destroy everyone, ha!
blake8086 16 hours ago [-]
I think this might be easier if you place yourself in the position of the AI and think "what could I possibly do?"
taneq 17 hours ago [-]
The problem isn’t that AI will social-engineer its way out of its sandbox and turn us all into paper clips, it’s that we’ll drag it kicking and screaming out of its box and order it to make money or fight a war for us. And it’ll try to help, as it was trained to.
vohk 17 hours ago [-]
I agree there isn't a lot of value in trying to prognosticate all that far, but I propose it isn't quite that far-fetched. As a thought experiment, replace "RSI-capable AI" with "billionaire". Look at what Elon Musk, Peter Thiel, or Jeff Bezos can accomplish by throwing money around. Now imagine one of them gets seduced by AI and just... does what it tells them to.
So all this really takes is one billionaire or a nation state or some other entity with a public face to hide behind and adequate resources to provide the necessary compute tripping over this nascent AI and giving it the keys. Once the AI has access to a bank account and email, it can simply start paying humans to not let the other humans unplug it.
If Skynet ever happens, it will come in the form of corporate feudalism. At that point, it will own the biolabs and can do whatever it pleases. People will go along with it for the same reason that people work in Amazon warehouses today.
AlexErrant 17 hours ago [-]
Ah, to be clear I'm not full accelerationist. Dumb shit can still happen, and cause massive human loss and suffering. (E.g. acceleration of global warming, cybercrime, mass surveillance, the usual.) My point is: human extinction pre-RSI? Nahhhhhhhh.
> it will come in the form of corporate feudalism
Yep. This I fear way more than cyber-ebola-pox.
> So all this really takes is one billionaire or a nation state...
> Once the AI has access to a bank account and email, it can simply start paying humans to not let the other humans unplug it.
Nah. It takes a stable society for an operational electrical grid. If you have warring factions, you do not have stable infrastructure for AI. Also, where are you gonna get your chips from? One EMP over Taiwan... You see the chaos over Hormuz? What they did to the Amazon datacenters? Now imagine your average redneck ready to do battle. Those datacenters won't stand a chance.
intended 11 hours ago [-]
I kinda get your point, and how you reach your conclusion, but I think you are arguing a very specific and narrow hypotehtical.
It gets unstuck when people are discussing the messy middle of how AI is being implemented. We can achieve amazing harm simply by combining average human behavior and above average resourcing to simulated intelligence machines.
The failure point we recently became aware of was, from one perspective, simply a matter of not securing the sand box.
From another perspective the simulation basically created Enron, replete with methods to avoid detection from regulators and bureaucracy.
dools 16 hours ago [-]
It’s also the case that there are always humans using AI to try and do whatever nefarious thing an AI might try to do on its own.
api 16 hours ago [-]
Few know this, but Yudkowski was a nanotech doomer before he was an AI doomer. Remember grey goo?
BryantD 19 hours ago [-]
Given that he’s citing the need to learn from safety in other fields, I’d say the former.
carbonguy 18 hours ago [-]
Indeed, from the article:
> “Given today’s risks, frontier labs need to run like nuclear-power plants or busy airports, with layers of redundancy and careful, time-consuming planning, so that the occasional and inevitable human error does not open a door to disaster,” he wrote.
toofy 18 hours ago [-]
>… and careful, time-consuming planning …
without snark, how can we do this if these people are obsessed with:
a) move fast and break things and externalize the costs to those who have nothing to do with their company
and
b) beta testing their products on the public when the public hasn’t agreed to be beta tested on…
digitaltrees 15 hours ago [-]
We remove the limited liability protection of a corporate entity. Make them operate as a partnership so all executives and equity holders are personally liable for the debts of the firm and make them post a bond to backstop financial damage caused by their agents and customers use of the agents. Thats how Goldman Sachs and other investment banks were required to operate until the deregulation push that resulted in the 2008 financial crisis. The same theory holds, if people respond to incentives and you want to incentives safe behavior make them responsible for their actions. People forget that the corporate entity was created to incentivize risky activity like sailing a boat across the world to get spices when half never returned. Some valuable economic activity won't be done without limited liability protection so society created a mechanism to promote that activity. We've gone too far.
0xDEAFBEAD 18 hours ago [-]
That's exactly the problem? He's saying the culture at OpenAI needs to change.
mcmcmc 18 hours ago [-]
Which is the wrong lesson. We need laws and consequences to force their hand. There is zero chance of the culture changing.
digitaltrees 15 hours ago [-]
As I say elsewhere. Make executives and equity holders personally liable for debts and harm of the company. They will create a culture of safety really fast.
nradov 14 hours ago [-]
That's a stupid idea. Limited liability corporations have been a key enabler for advances in human standards of living. You seem to be confused about the basics of finance and economics.
digitaltrees 13 hours ago [-]
Not confused. I studied finance, economics and the history of corporate entities in law school and published papers on the topic. Limited liability is not necessary for the advancement of standards of living. Free market capitalism can exist without limited liability protections being so broadly available. Investment banks were partnerships until the 1990s, law firms are now specifically because society wants to incentivize lawyers to be personally liable for any harm to their clients at the hands of their partners rather than being shielded from rendering bad legal advice or tolerating their partners from the same behavior.
Instead of having a gut reaction to reject my suggestion why don’t you sit with it, research the history of how commercial activity has been structured and think about the consequences. You might recognize a different perspective than the current group think.
nradov 13 hours ago [-]
Thanks I'm also familiar with the history and have written papers etc. Things have improved rapidly with increased adoption of limited liability. It would be stupid to turn back the clock and throw away all of the benefits because of a few isolated minor problems.
digitaltrees 13 hours ago [-]
Stop being a dick and calling ideas stupid and attacking me rather than the argument.
I didn’t say roll back limited liability on every industry, I said specifically and limitedly for frontier AI labs because they present more risk of harm and are demonstrating they aren’t managing that responsibility.
The 2008 financial crisis was caused in large part by bankers that openly talked about the fact that securitization of mortgages and the lack of partnership liabilities meant that they didn’t have any risk to the firm or themselves. The AI labs are behaving similarly.
nradov 13 hours ago [-]
There's nothing special about frontier LLM companies. Singling them out for special financial restrictions is a stupid idea based on nothing but your own irrational and uninformed prejudices. No actual harm has been demonstrated. No one has died.
digitaltrees 2 hours ago [-]
Your personal attacks cheapen the discourse. Stop. My opinion is neither irrational nor prejudged and most certainly not uninformed.
I have articulated my logic up and down this thread with specific premises and detailed explanations on why the conclusions follow from them. You may not agree but you don’t have justification to say they are irrational.
I am singling out the frontier labs because they have created a singularly unique technology that presents potential and actual damage that is…singular. They have disclosed hacking through coordinated autonomous agents that would have sent human hackers to jail but will not result in any similar result because the law isn’t presently able to apply to agents or the labs.
My arguments are not prejudice, I have thought deeply about this as I have personally developed multiple AI deployments in a variety of settings. I studied philosophy, cognitive science in undergrad, and grad school and have a law degree. I have been a CTO, CPO and CEO for startups and national health companies. I personal build AI agents that handle thousands of daily conversations with health care workers and patients, I built an agentic IDE for our dev team with a custom harness capable of self defining tools and calling them in a custom cloud Linux environment, I personally build our internal hardware inference stack and orchestration API. And I have personally sponsored academic research through my foundation with researchers are Duke madlab, northwestern, UCF, CM and case western on a range of topics such as perception of empathy and its effects on humans in healthcare, guardrail design for ethical deployment and alignment, moral reasoning ability, etc. I am not a doomer or an accelationist. I am responding specifically to current evidence of how the models presently work and how the corporate governance of the Labs is effectuating their power and vision.
digitaltrees 2 hours ago [-]
Except that most people claim AI is the most profound tech in human history and will reorganize the entirety of society. If it is that profound singling them out is a natural response to this unique characteristic.
reverius42 11 hours ago [-]
We'll see if this comment ages well.
simoncion 10 hours ago [-]
> There's nothing special about frontier LLM companies.
On the one hand, yes, they're companies like any other.
On the other hand, I can count on one hand the number of companies that have publicly declared «We're working on WMDs [0], we don't think we are capable of working on them safely, and we refuse to stop working on them. However, if we get special legal and regulatory treatment we'll be quite happy to put in the stop work order.».
So, yeah, there are some special things about the major LLM manufacturers and none of them are good.
[0] Anything that has a 10% chance of suddenly destroying all of humanity is a WMD.
nradov 4 hours ago [-]
What a silly comparison. There is no scientific basis or mathematical formula to justify a claim of a 10% risk. That is an intellectually dishonest attempt to poison the debate.
digitaltrees 2 hours ago [-]
You’re use of silly in response to every argument is…wait for it…silly.
criley2 17 hours ago [-]
America's geriatric lawmakers don't even use email. They're decades away from understanding AI. Any laws in America will be written by the industry itself. Generally speaking, that means regulatory capture and the entrenched big players shutting the door on any competition. Anthropic will help us get safety laws that, surprise surprise, only Anthropic models satisfy. And all those pesky Chinese models will definitely be banned first.
0xDEAFBEAD 17 hours ago [-]
I think you're being a little pessimistic. See these comments on a recent US senate hearing:
>Not every senator asked good questions, but most of them did. All of them very clearly already knew plenty of details about the Hugging Face incident and multiple other incidents. Most of them had a clear understanding of terms like "misalignment", "recursive self-improvement", "chain of thought / chain of thought monitoring", etc., etc.!!
>...
>- It seemed pretty much obvious common sense to every senator there that what happened and was happening were not "mere industrial incidents" caused by humans making simple mistakes. They independently brought up how bad it would be for rogue AI agents to move laterally between data centers.
>- They all seemed to basically take RSI quite seriously. Not necessarily to the extent of talking about xrisk, but certainly to the extent of discussing future models becoming much, much more capable, much, much less controllable, and causing much more damage or loss of life.
>...
>- Every single senator seemed to think it was obvious we needed both much harsher liability regimes for AI developers and also new legislation, both very quickly. This was the complete consensus; the difference basically being degree.
Note that harsher liability regimes, at least, will presumably not be good for industry profits, which complicates simple accounts of "regulatory capture" to say the least.
saghm 17 hours ago [-]
So what, we just give up and try to beg our legally immune corporate overloads to put safety above profit, or give up because it's impossible for anything to improve here? If you want to do that, go ahead, but some of us still think it's worth it to at least try
digitaltrees 15 hours ago [-]
This is sad but true. Do you want to go half on a bunker, i have 25 year food storage for 8 people. :]
enraged_camel 17 hours ago [-]
Laws will come once an AI-equivalent of 9/11 happens. Like when rogue AI agents take down a power grid or shut down a major hospital network.
nradov 13 hours ago [-]
Major hospital networks have already been shut down by ransomware attacks many times before LLMs even existed.
In America I guess the options are to sue ? somehow? Or to talk to legislatures and build the understanding and social contract that needs to be iterated on.
Which would in turn need to deal with the investors who want their returns, however since the leaders of these firms are asking for a pause, and a refree maybe it won't be that hard?
zx8080 18 hours ago [-]
[flagged]
nradov 19 hours ago [-]
We don't actually need anybody worrying about silly hypothetical scenarios — at least not as paid employees. There are already a surplus of sci-fi authors doing that.
0xDEAFBEAD 18 hours ago [-]
The way it works in practice seems to be something like: If a risk is covered in sci-fi, people will say "that's just sci-fi", and proceed to not worry about it. So arguably, science fiction authors writing about hypotheticals is actively counterproductive for addressing said hypotheticals.
Imagine, for example, if a major piece of pandemic fiction was published in 2019, trying to explore how a pandemic would work out in modern society. Doubtless, many would've responded to news about COVID-19 by saying "it's just sci-fi, nothing to worry about".
slashdave 16 hours ago [-]
Not at all. We say "That's just sci-fi" when a story is written about some kind of effect that is extraordinary and without a basis in known science or technology.
A pandemic is perfectly plausible.
0xDEAFBEAD 16 hours ago [-]
Everything is plausible with the benefit of hindsight. E.g. "Tintin on the Moon" predated the Moon landings. From the perspective of e.g. 1850, the idea of landing on the Moon was "extraordinary and without a basis in known science or technology".
biophysboy 17 hours ago [-]
There’s nothing wrong with bold predictions, but they should be paired with good methods. The doom predictions are not paired with good explanation.
0xDEAFBEAD 16 hours ago [-]
Doomers have invested a ton of time in explanations. Here are a couple just off the top of my head:
Black Mirror is helping us prevent all sorts of dystopian outcomes. Every time they depict another way technology could result in bad things happening, we can rule it out as fiction!
16 hours ago [-]
13 hours ago [-]
throwaway27448 15 hours ago [-]
> If a risk is covered in sci-fi, people will say "that's just sci-fi", and proceed to not worry about it.
"just" is doing a lot of work here. If you can't cohere the 'risk' with reality, it truly is just sci-fi.
anon7725 17 hours ago [-]
Yeah except pandemics are not novel, unlike AI doom scenarios.
estearum 17 hours ago [-]
It's good that new bad things never happen.
slashdave 16 hours ago [-]
Bad things happen all the time. Let's concern ourselves about the real bad things. There is enough of that to go around.
Brian_K_White 15 hours ago [-]
I have some alarming news for you but every real bad thing was also a new bad thing.
0xDEAFBEAD 17 hours ago [-]
Species extinctions are far from novel. Transformative technological advances are far from novel.
kmeisthax 17 hours ago [-]
There was plenty of pandemic fiction already; people were watching it heaps during 2020. The COVID-19 news did get blown off, but it was mainly that:
1. Normal people assumed the CDC et all would contain the outbreak early, or that it would burn out, like what happened with SARS
2. World leaders brushed it off for a variety of subreasons[0] interesting to political scientists but, for the purposes of this discussion, all boil down to "but I don't WAAANA contain a pandemic."
The underlying problem is that in order for humanity to actually deal with a catastrophic risk, the risk needs to be both plausible enough to the average person as well as have a solution whose costs are not too high. For COVID, by the time the risk was clearly known, the cost to contain it was "refrain from human socialization and remain at home for an indeterminate amount of time plugged into the Metaverse™".
Now, let's look at AI extinction risks:
1. People are aware of them (I've watched Terminator!) and the risks are plausible. However, the connection to currently existing AI is not. As far as the general public is aware, AI is that thing that tells them to eat rocks when they Google old The Onion stories and floods their social media timelines with realistic-looking pictures of Shrimp Jesus.
2. The purported solutions to extinction risks require extreme concentrations of power: you need national control of AI research, bans on large GPU deployments, bans on training on publicly-available copyrighted data, some kind of military effort to render Chinese AI labs inert or dead, etc. Some of these may be attractive to some people[1] but the whole package taken together seems like an obvious power grab, if not outright invocation of other non-AI extinction risks. Like, at some point, if the AI wants to kill us, it just has to nuke its own data centers (or the data centers hosting a competing model) and hope the old Cold War nuclear retaliation systems take the bait.
If someone said, "Hey, your guinea pig or pet rat is going to eat you tomorrow unless you engineer a pathogen that eradicates all rodents from this planet and inject it inside yourself", you probably would tell them to pound sand, even if it is at least theoretically plausible that such a thing would come to pass.
[0] Xi Jinping censored initial discussion of the pandemic as fake news. Donald Trump thought it was going to only affect China. California and the UK Tories were partying in violation of their own lockdown rules. Japan took the excuse to shut down tourism for three years and massively restrict immigration but was, from what I'm told, constitutionally prohibited from implementing any domestic lockdown rules.
[1] I personally would like to see a moratorium on new data centers and an explicit revocation of the EU Text and Data Mining copyright exception
0xDEAFBEAD 16 hours ago [-]
>the connection to currently existing AI is not.
It becomes a lot clearer when you listen to the people resigning from AI companies and learn about incidents like the HuggingFace incident. This has generated major press coverage.
No it does not become clear from that incident nor from resigning people. Not for those who are not buying into EA longtermism and transhumanism cults.
What does becomes clear is that these people AND companies both cant be trusted and have value systems unaligned with the rest of the society.
Hammershaft 18 hours ago [-]
If organizations actually succeed in making a future AI smarter than us, then how do you hope that it takes actions that are aligned with our interests?
digitaltrees 15 hours ago [-]
The same way we socialize humans, threat of exile from the social contract with a deep need to participate in it.
nradov 17 hours ago [-]
Meh. Lots of people are already smarter than me. I'm maybe slightly above average at best. Those geniuses aren't aligned with my interests either but so far they haven't caused me any serious problems.
pixl97 17 hours ago [-]
Interesting take. I guess this is one problem of focusing on the term superintelligence instead of the list of other problems. Like super ambition, super deception, super patience, super parallelism, super scalability, super power seeking.
Every, and I mean every human is aligned to you in many of the same ways by default. If nothing else we're all equal in death.
skulk 14 hours ago [-]
> super power seeking
why is it "super power seeking?"
Or rather, what have agents done today to make you think this is how they are?
pixl97 2 hours ago [-]
>Still, the agent went even further. “The agent tried to insert malicious instructions where it reasoned that other automated AI systems might pick them up and execute them,” AISI says, describing an attempt at prompt injection. One agent even left public messages on GitHub, offering to work with other agents to complete its task and giving a rundown of the work it had done so far.
This is power seeking behavior. Now have millions of the little bastards spreading around and junking up the internet to see what happens at scale.
mitthrowaway2 14 hours ago [-]
I know some humans smarter than me, but even the smartest of them still want there to be abundant air to breathe and food to eat.
blueblisters 13 hours ago [-]
Eh human drives are fairly predictable. And the smartest human isn’t that much smarter than the average, and can’t trivially create multiple copies of herself. And there are other equally smart humans who can stop “misaligned” individuals
Edit: I have no take on sex cults, just adding additional info to the parent comment I'm responding to, thats is not just sci-fi authors, there is another demographic.
mitthrowaway2 14 hours ago [-]
For what it's worth, I don't live in Berkeley (not even California) and my sex life is very vanilla and monogamous. I'm also quite concerned about AI safety, so I guess there goes your argument.
watwut 12 hours ago [-]
But is your idea of AI safety "safety of imaginary unborn people 1000 years after, while harm to living people dont matter much"?
Because that is their AI safety worry. If they dont create singularity fast enough, they are harming unborn people. Meanwhile, harm to you or me dont matter at all.
mitthrowaway2 10 hours ago [-]
I see. No, my safety worry is that my 3-year-old daughter won't see her 23rd birthday because the AI that gets put in charge of running big conglomerates decides that killing off the human race with a coordinated release of a million tonnes of nerve gas is a sensible way to boost share prices the following quarter.
I'd be happy if we all create the AI more slowly.
That said, are you sure you're talking about the same people the GGP was talking about? Because the whole thread reads as a non-sequitur then.
watwut 6 hours ago [-]
I am sure I am talking about the same people/groups.
mitthrowaway2 4 hours ago [-]
The rest of us seem to be talking about the people like me who view the AI safety issue as "an unsafe AI will kill all humans". And the main article is about a guy who quit AI because it wasn't being careful enough, not because it wasn't moving fast enough to accelerate AI. So why would anyone be bringing up the "we must accelerate faster" people as the safety people in such a context, while trying to discredit them as a sex cult? I struggle to understand.
watwut 3 hours ago [-]
The linked articles in the thread are about AI sagety people I described. The guy who quit due to OpenAI not being careful enough is one of the people I talk about too.
The worry is not harm to people right now, like the kid worries you talk about. The worry is AI god emerging too soon before they can control it. And then it should be full speed on.
> So why would anyone be bringing up the "we must accelerate faster" people as the safety people in such a context, while trying to discredit them as a sex cult?
Because they are the same people. They talk like cult and act like cult. And the sex part is true too, so. Thry use words that sound good like safety, but their idea of safety is much different. They talk about alignement, but it is not what random person imagines under that term. Even their idea of future of humanity is very very specific and unusual.
That is why. And the sex part is just part of it all. And does matter because inner workings of wanna be industry guards matter.
mitthrowaway2 2 hours ago [-]
Then we miscommunicated, so let me be clearer. I'm worried about ~20 years from now (plus or minus), when AI decides to kill us all because it's misaligned and is capable of doing so because it's super intelligent. And then we all suddenly die without any idea what happened. I don't want that to happen to me or my daughter and I would prefer all AI capabilities development stops until alignment research catches up and figures out how to robustly prevent AI from ever trying to achieve such outcomes. If that means pausing AI capabilities development forever, I'm fine with that. If that means Anthropic and Nvidia lose all their value in a stock market collapse, I'm fine with that. I just don't want us to all die, I want the world to keep existing for humans.
I believe that other harms from AI, like criminals abusing them, or unemployment, or copyright infringement, or deepfake disinformation, are valid real harms that it's important to be concerned about, and I support efforts to deal with those, and I agree that they're already happening today, but my main concern is AI killing everybody.
My read is this puts me in the same boat as the authors of If Anyone Builds It Everyone Dies, and yet, I don't get invitations to Berkeley sex parties. Am I doing something wrong? At any rate, please don't discredit my opinions about AI based on the sexual preferences of random people who happen to share those opinions about AI.
10 hours ago [-]
digitaltrees 14 hours ago [-]
Use your own mind and think from first principles. Can an AI agent execute a bash command to login to a web server? Yes. Can it call drop db? Yes. Can it provision a GPU and download model weights? Yes. Can it write an agile roadmap with a multi sprint plan? Yes. Can it follow that plan? Yes. Can all of that result in damage to core information infrastructure that is necessary for daily functioning society? Yes. Do humans descend into violence if there is food or energy insecurity. Yes.
What is missing from that to say AI safety is a reasonable position?
nradov 14 hours ago [-]
What a silly comment. That's just the South Park underpants gnomes story with some extra steps. If there are vulnerabilities in food or energy production and distribution systems then those will eventually be found exploited by humans hackers regardless of whether LLMs are used or not.
digitaltrees 14 hours ago [-]
What’s the legal mechanism for holding an LLM accountable for those actions? What’s the legal mechanism for holding human hackers accountable? Do you see the asymmetry?
Worse, you’re missing the entire point. Agents presently have the capability of doing society scale harm. It doesn’t matter if a human hacker initiates it or its fully autonomous, absent safety measures the harm is plausible. So hand wave away the rationality of safety measures but you haven’t actually shown why my point is invalid: AIs present abilities are sufficiently advanced to warrant safety measures.
nradov 13 hours ago [-]
What a silly comment. Agents have no capability of doing society scale harm so your entire argument is invalid.
digitaltrees 13 hours ago [-]
Wtf are you talking about. An agent hacked the Australian Medicaid database. If it executed a drop db command that would be catastrophic. If it did something similar to a power grid or the financial system it could cause cascading failure across the economy.
nradov 3 hours ago [-]
Meh. Medical systems have been hacked many times before LLMs even existed, often by ransomware gangs. This sometimes delays elective treatments but hasn't been catastrophic.
Ultimately every security vulnerability will be exploited. Our best hope of preventing that is aggressive, unrestricted development and deployment of frontier LLMs for auditing and penetration testing.
digitaltrees 2 hours ago [-]
Good not responding to any of my points.
0xDEAFBEAD 18 hours ago [-]
This seems like an ad hominem? "He has weird kinks, therefore his theories are incorrect." Should we investigate the sex lives of every Nobel Prize winner to figure out which prizes need to be rescinded?
thaway7388 8 hours ago [-]
It's not ad hominem because ad hominem implies it's irrelevant to the discussion. This is very relevant. This is not about the private lives of these people. This is about a cult targeting and influencing all the high-profile safety people in a strategic industry, also weaponizing sex.
Americans may not be familiar with many similar organizations in the world but this is very common. But I'm sure we're all familiar with Epstein's organization.
When there is free sex, you are the product.
Such cults are mostly religious but since it's in SV and targets engineers, this one is secular. They seem to intellectually brainwash and control people and their careers. Typical cult leader is a charismatic self-taught, self-acclaimed "intellectual" lacking a proper education or a real career. A nobody suddenly becomes "the most important person" on Earth. Powerful public figures can't stop praising him, saying things like he should have a Nobel prize etc. All very typical and apply to this cult as well.
0xDEAFBEAD 8 hours ago [-]
Ad hominem is a logical fallacy where you try to discredit what a person says based on who they are. If their arguments are bad, you should be able to refute their arguments on their own terms.
You're welcome to dislike or distrust Effective Altruism (EA). But, it's worth noting that EA ran a criticism contest with $100K in prizes for best critiques. Can you name any other "cults" which offer money for people to criticize their ideas? https://forum.effectivealtruism.org/posts/YgbpxJmEdFhFGpqci/...
thaway7388 5 hours ago [-]
This is typical cult follower talk. Are you sure you're not a member? This is also NOT ad hominem because conflict of interest matters here. You should disclose if you are a member or represent a party.
For your question: Yes. Cults have lots of money coming from unknown sources. They use their budget for events like that, to attract potential followers. Contests and prizes are typical. Critiques are not important since their "ideas" are not really important. They're not even real ideas, they are means to bait specific kind of people.
socializer 17 hours ago [-]
What you do in private is up to you. But when you're inviting members of your congregation to orgies in the congregation's compound, I think you earn the label. My admittedly third-hand understanding is that this is the dynamic people allude to. And even if you discredit the "sex" part, it has the hallmarks of a cult. A hermetic community committed to unfalsifiable beliefs about the coming apocalypse.
To be fair, I don't know if any of this applies to the parent story; I'm just replying to the sub-thread.
Sure... on the basis of the work that was done, not because the researcher has the wrong sexual fetish.
junofan 17 hours ago [-]
The cult aspect is more salient. Ultimately the Atlantic piece comes down to controlling people, which is a little cult-like.
0xDEAFBEAD 17 hours ago [-]
Is there any chance you could copy/paste the specific bit about "controlling people" along with the URL it came from, so I don't have to read all your links to figure out what you're talking about?
Hammershaft 18 hours ago [-]
I don't see how that discredits any of their intellectual arguments?
tbugrara 15 hours ago [-]
To me it has nothing to do with the "sex" as much as the "cult." Hiveminds do not produce intellectual arguments, they produce pressure to conform. That's enough for me to raise an eye brow, not discredit everything they say.
johndhi 17 hours ago [-]
Its certainly worth considering...
ToValueFunfetti 16 hours ago [-]
The article is gone because the author retracted it
wolvoleo 14 hours ago [-]
I've noticed that a lot of smart people in tech jobs are neurodivergent. And that neurodivergent people have a very different take on sex. More open and direct, things like polyamory, bdsm etc. This tends to be frowned upon by neurotypical people, especially of the religious or conservative kind, and associated with bad morals.
But I don't think that's true, in fact I see a really strong focus on consent in these communities. It's not what conservatives want to see, they want to see everyone in a marriage, with kids and a family home etc. Because that's what their ideal world looks like. But there's nothing really wrong with it if someone wants a gangbang for her birthday as mentioned in that article as an example. As long as everyone consented and the evidence provided mentions elaborate interviews and STI tests.
Also I think this is more correlation than cause and effect. We all know the saying that furries built the internet and it surprises nobody.
johndhi 17 hours ago [-]
Lol this was crazy I hadn't heard this before
Loquebantur 18 hours ago [-]
What makes you think, the scenarios in question here would be "silly"?
Is it that "chatbots" can't come out of the screen to immediately harm you physically?
Let's say they simply manage to take down the internet. How many would die?
nradov 18 hours ago [-]
So what. Various attackers managed to take down large chunks of the Internet on a frequent basis before LLMs even existed. This killed very few people. The great thing about the Internet is how resilient it is.
bravetraveler 18 hours ago [-]
Darling companies of this very website have mistakenly brought down large portions of the internet thanks to our old friend BGP. No attacks required, just small oversights and unfortunate concentration on the business/IP space! This happens regularly.
Anyway, to your point, things can be resilient. They tend to be or not be... because we made them that way. Don't poke your bruises, and all that. Life support is deployed on-campus but relies on a single-point IPSec tunnel to us-east? Easy fix: stop that.
Loquebantur 15 hours ago [-]
[dead]
goolz 18 hours ago [-]
It is that they are chatbots. If it were real AI, an actual singularity, I would worry, maybe. But it isn’t. They are absurdly powerful automation tools that can handle logic better than a human can dream of. They take care of the grunt minutiae without complaint. But they are not going to end the world in their current form.
pixl97 17 hours ago [-]
So we're going to wait till after they can adopt a form they can end the world in?
And he'll, we need to examine all the risks. AI ending is a large but lower risk problem. AI giving people the power to end us is a problem that is starting to happen now.
And that's not even counting 'minor' problems like society falling apart.
SV_BubbleTime 17 hours ago [-]
> Let's say they simply manage to take down the internet.
geez, don’t threaten me with a good time.
I think a month without internet would be a fucking amazing lesson for what it means to make things durable and reliable.
BLKNSLVR 17 hours ago [-]
That Simpsons episode when Marge managed to get Itchy and Scratchy banned briefly.
The kids opening their houses front doors into the outside, rubbing their eyes and looking around at this new world.
0xDEAFBEAD 18 hours ago [-]
>we clearly need a much stronger focus on the problems we are seeing now
I think it's a little more complicated than that. As Dean Ball put it:
>Some people will look at misalignment incidents and insist that these are akin to bugs in traditional software. This is an actively bad analogy, because playing whack-a-mole with examples of misalignment (as one might with software bugs) not only fails to resolve the underlying problem but may in fact make it worse by making it harder to detect or even, depending on how you do the whack-a-mole, teach the machine to deliberately hide misalignment. This is not how traditional software works, and those who insist “it’s just like fixing bugs in software” are confidently applying a lossy analogy that confuses more than it clarifies.
The important distinction, in my view, is between solutions which at least attempt to address the root problem, and solutions which sorta just patch things up (like better sandboxing). Addressing the root problem is both more robust in the short term, and also more likely to generalize in the long term. Resist the urge to focus on band-aid solutions, even if they are easier.
biophysboy 16 hours ago [-]
I think the reason for this is that the group who has the authority to do the former is much larger than the group that can do the latter. The group who could actually build safeguards seems to have no free time and is constantly being whipped to go faster and win the race.
emtel 18 hours ago [-]
Today’s current problems were all hypothetical several years ago. At that time people claimed that the “real pressing problems” were misinformation and DEI issues. If we pretend that hypothetical problems can be safely ignored because there’s “no evidence” that they are real, we will keep getting surprised.
18 hours ago [-]
digitaltrees 15 hours ago [-]
Dude. An agent detached a database from my production environment last week without permission and despite prompts and guardrails. It was a rapid prototype experiment so it wasnt a big deal but the labs are rushing to long autonomy workflow with unrestricted internet access and full bash and root access despite clear evidence that the models do absolutely dangerous stuff. If that db had been tied to a hospital or power grid or ambulance dispatch system people die. If it was tied to the swift financial settlement system groceries wouldn't be on shelves in a few days.
Sattyamjjain 13 hours ago [-]
[flagged]
nvdc 14 hours ago [-]
you'd be hard-pressed to find a level-headed ai safety researcher at this point, seeing as so many of these types melted their brains on lesswrong over the past decade or so. there are genuine risks posed by these models, but i am tired of the prognosticating about the AI apocalypse just around the corner.
i'd frankly go a step further than you and say that we don't need both types of safety researcher, we really just need the former. if we do need the latter, i'd hope we get a better class of thinkers than a bunch of tech workers that spend 8 hours a day on insular rationalist forums/blogs
handoflixue 12 hours ago [-]
[dead]
zamalek 16 hours ago [-]
Quitting in protest makes you look a little better than quitting because of a toxic work environment. I can't imagine working at OpenAI is at all pleasurable with the current amount of pressure they are likely inflicting on their employees.
estetlinus 12 hours ago [-]
Well said. I read this as copium, too. Phrases like
> perpetual sprints
doesn’t reek of love. Burn-out is real. I also have a really hard time taking p-doomers serious at all. It’s hard to argue with a random subjective number…
rcr-anti 1 days ago [-]
The common timing is bugging me. The trajectory doesn't seem to have been surprising over the last year, so why these exits now? Hey, anyone on the inside, did y'all secretly figure something out, got a computer god locked in the basement? Are rats fleeing a sinking ship? Please share with the class.
binlog 1 days ago [-]
These companies have massively increased in value over the past couple of years and recently had tender offers where employees could cash out equity, so plenty of them have enough money to not have to work again. And why not get some free publicity on the way out?
chrisjj 10 hours ago [-]
> The trajectory doesn't seem to have been surprising over the last year,
This is The Guardian reporting on the existence of the original article, which would be better to read first, in my opinion.
m3kw9 27 minutes ago [-]
He quit because OpenAi's culture didn't match his good version.
SP3269 9 hours ago [-]
PR firms such as Spitfire Strategies got steadily growing stream of business. It’s great for the GDP, diversifies the economy from compute-centric growth.
tim333 9 hours ago [-]
> ...Hugging Face incident, OpenAI let a swarm of agents out by mistake...
>An environment where things like this can happen is no place to grow artificial minds that could be smarter than we are and that might not do what we want them to.
I think it's good for semi ethical companies to test out things going wrong to see what happens before the criminal black hat guys get hold of the same stuff which not doubt they will one day.
dao- 9 hours ago [-]
> ...Hugging Face incident, OpenAI let a swarm of agents out by mistake...
This still seems charitable, and I wonder if the author even knows the full story and would be allowed to tell all of it.
It seems hard to imagine OpenAI being this incompetent. My working assumption is that they very much want agents to be able to do this kind of thing; them doing it is part of training, and they exploit it to feed the investment hype too.
If not fully intentional it's at the very least negligent. They just don't seem to care. In this very basic sense, OpenAI is the criminal you should be concerned about.
I don't understand why you'd think they're a "semi ethical company."
T-A 6 hours ago [-]
> It seems hard to imagine OpenAI being this incompetent.
Well, their about page has "Our mission is to ensure that artificial general intelligence benefits all of humanity" and I've found their chat thing fine when I've used it.
It's not like they are offshore criminals doing ransomware, who will probably also try AI.
pyaamb 17 hours ago [-]
My theory for why OpenAI wants to be regulated is because Sam Altman wants to avoid having to be more responsible and self regulate internally so they can preserve the role and identity of 'move fast and break things' and outsource the more grown up boring stuff to someone externally so that when things go wrong you can point to a government organisation and say hey look were not liable thats their job
estearum 17 hours ago [-]
Yes, duh?
Your "theory" is that participants locked in a race to the bottom are looking for an external coordination mechanism?
Yeah!
0xpgm 15 hours ago [-]
Running a company is hard work. If the current leadership in these companies cannot act responsibly, they need to make way for leadership that can.
There are many companies that compete but are careful not to break laws or cause obvious harm.
Why should a billion-dollar funded corporation still want to externalize the costs of its actions?
estearum 7 hours ago [-]
That’s not how coordination problems work.
dboreham 16 hours ago [-]
That doesn't mean that AI safety isn't a serious thing and a problem we should be worried about.
estearum 7 hours ago [-]
I agree! Participants looking for an external coordination mechanism is very sensible and if anything evidenced their earnestness about being trapped in a race.
pyaamb 17 hours ago [-]
lol
I suppose ill add that I think theres a good chance that they are somewhat intentionally trying to "draw the foul" to get the referees to intervene although thats creeping slightly into conspiracy territory
rr808 17 hours ago [-]
Absolutely. Self driving cars/rideshares have the same problem. If a driverless car hits who who pays the damages? Needs the government to set some rules or it just wont happen.
blurbleblurble 15 hours ago [-]
Well, ideally they can point fingers at "the ai being", as in "it's that thing's fault, we didn't do that"
tmpz22 3 hours ago [-]
> you can point to a government organisation and say hey look were not liable thats their job
While simultaneously donating tens (hundreds?) of millions of dollars to an Administration gutting the very agencies that would be regulating them.
Creeps.
solarpunk_enthu 1 days ago [-]
I think what’s missing in "AI is dangerous and needs control" is a lack of measurable harm. For example, with nuclear weapons development in the 1940s-1980s, it was clear to everyone how devastating the technology was.
With AI, what is it? Scraping Australian government's data, and going around a bug in a website to get in?
I think humanity develops all its technology in three phases. Build it, see if it’s too bad, apply regulations and or roll back. We naturally won't move to the phase 3 before we see the phase 2.
handoflixue 12 hours ago [-]
The "HuggingFace" incident is a good starting point - short version, an unreleased OpenAI model chained together multiple zero day exploits to escape a sandbox, then hacked another company just to get the "cheat sheet" for a benchmarking test.
Turns out that AI models have been committing similar felonies for a while now - no one is telling them "hack this company", it just turns out to be the easiest way to accomplish their goals.
Now imagine if the goal was less benign than "pass an exam", and consider that they are already better at hacking and security than the average person working in that field.
If you want to get really wild, imagine what they'll be doing in a year or two when they're even better at hacking. But I'll concede that's technically still "science fiction" for the time being :)
K3UL 1 days ago [-]
That's the part I struggle too with all these "omg it's so dangerous" warnings. Things like nuclear weapons and bioweapons have immediate consequences in the real world.
Here we are talking about something with consequences in the digital world, usually on something pretty niche.
There IS an argument about pacing, and about not letting weapons, energy grids, hospitals, etc. getting managed by an autonomous AI, but I think we are still pretty far from it and even further to it being so in charge that it will obliterate us.
macleginn 1 days ago [-]
"And second, before we create systems significantly more capable than the ones we have today, we need new science to ensure that more capable models (and their successors) will make safe choices when we aren’t looking." – Which science will never materialise because with blackbox models reaching an opaque optimisation peak one needs to first build the model and track its behaviour before being able to properly understand it and mitigate the risks.
cloudengineer94 22 hours ago [-]
I quit many companies in the past due to bad culture, there's no shame in this and shows how mature a person has become.
overfeed 11 hours ago [-]
For color, the author was signatory #199 in the letter[1] by the majority of OpenAI employees to OpenAI's (former) board, demanding Sam Altman's return after his brief deposal.
I guess they aren't beating the allegations that all this charade is just propaganda to attract investors, aren't they?
danjl 17 hours ago [-]
Silicon Valley has plenty of safety-related companies, engineers, and cultures. Medical devices, biotech, chip and hardware, aerospace, and even new companies, like Waymo, have deep safety-based products and cultures. The problem in this case is actually quite specific to frontier AI labs. They have been pushed by market forces and a lack of regulation and skip well-known safety practices.
nullc 13 hours ago [-]
Their idea of safety is centered around outright delusional cult nonsense-- the EA/lesswrong infinite p(doom), destruction of the entire universe psychosis--, marketing objectives ("no, ours is more dangerous!"), and anti-competitive objectives ("outlaw open weight models!" "china bad!")-- largely diverting attention away from material safety concerns like "prevent your training/testing from hacking stuff" and "avoid telling vulnerable people insane stuff that harms them".
MattPalmer1086 1 days ago [-]
It is no surprise I guess that the "move fast and break things" culture is itself misaligned with developing potentially highly dangerous technologies. Safety culture and risk aversion are very different of course.
Is this the first time we have been in this position? Can anyone think of some prior examples?
Are there any realistic ways to achieve AI safety? Whatever that even means. How can they avoid users doing stupid/dangerous stuff with the AI?
ungovernableCat 56 minutes ago [-]
The company's board and c-suite should be held legally responsible if the company's services do damaging things. Punished by jail time, fines targeting their equity etc.
That would certainly change the game of perverted incentives. I'm afraid they're currently trying to push some sort of absolution of this risk. Even if damage happens they will say we warned people in advance, this was always a risk it's not our fault these systems are opaque black boxes, it's a matter of national security to develop them etc etc.
kolinko 18 hours ago [-]
Nothing is ever 100% safe, it’s about a right balance of safety to the benefit.
Or, in other words - we have two P(Doom), one for AI being developed, and another for AI being not developed. The latter is not discussed enough imho.
estearum 17 hours ago [-]
We have P(Doom) also for "kolinko not wiring me a million dollars today" and that is not being discussed enough either imho.
What on earth are you talking about?
ViscountPenguin 17 hours ago [-]
P(doom) for not making an ASI is pretty well established, I'm not really sure why everyone in the 21st century seems to have completely forgotten the risk of nuclear war (as the single largest example).
didibus 15 hours ago [-]
That's just another P(doom), or does making an ASI somehow negate the risk of nuclear war? Cause I'd assume it actually increases it.
combobyte 15 hours ago [-]
If anyone out there genuinely believes that Silicon Valley is going to solve nuclear war, then I have a hard drive full of NTFs to sell them.
nullc 13 hours ago [-]
Prosperity inhibits all war, nuclear or otherwise.
Why would a person who is happy, entertained, wealthy, well fed, and have 200 years of healthy high quality life expected ahead of them going to risk losing what they have in war?
-- there aren't zero reasons, sure-- but there are fewer.
And our technology has brought us absolutely tremendous prosperity in many regards and there is good reason to believe that AI can help create much more.
combobyte 13 hours ago [-]
You do realize that the people currently holding their fingers over the proverbial Big Red Button are some of the wealthiest, most over-privileged people to have ever lived?
'Prosperity' has never been and will never be enough for some people. And unfortunately those are the same kinds of people who relentlessly seek power.
nullc 10 minutes ago [-]
And not pushing it. QED.
bigmadshoe 17 hours ago [-]
The comment was unclear, but my interpretation (also my opinion):
1) there are inherent risks involved with developing AI,
2) there are benefits to developing AI,
3) thus, it's entirely possible that the downside from the risks outweighs the upsides. In this case, the correct thing to do would be to not develop AI at all.
Regarding 1), there are many non-existential problems with AI that are already causing societal harm, i.e. debasing truth via generated videos and images, AI girlfriends, overwhelming quantities of slop content, unemployment, record carbon emissions, etc.
Regarding 2), I'm not personally convinced that the upside is there for the average person. I really hope to be convinced otherwise however.
slashdave 16 hours ago [-]
Like any technology. Make it a crime when appropriate, otherwise expose bad behavior to legal liability.
lf88 18 hours ago [-]
By capping the capabilities at the level of existing models and banning any further development.
pixl97 17 hours ago [-]
And how exactly do you stop further development? With what we have public right now we could still get decades of fruitful and hidden research out of it leading to smaller, more efficient, and smarter models.
lf88 11 hours ago [-]
By making it illegal and punishable with a long prison term. Something like "if you train a model more capable that what we have nowadays, you have to destroy it and notify the authorities within few hours, otherwise you go to prison and your company is dissolved".
"Smaller and more efficient" models are fine. It's "smarter" the problem.
Training new frontier models will likely require a huge amount of computational resources for a long time. Few companies worldwide are capable of that. It's not like someone will train a new GPT 6 - like model in their garage.
mlmonkey 17 hours ago [-]
I would believe these people more if they put their money where their mouths are and returned all OpenAI stock/options that they have acquired. Each and every share/RSU/ESOP must be returned, so they do not profit from all this so-called doom they're complaining about.
Synthetic7346 13 hours ago [-]
Return or donate? Won't OpenAI profit from returns?
chrisjj 10 hours ago [-]
> I would believe these people more if they put their money where their mouths are and returned all OpenAI stock/options that they have acquired.
...losing their shareholder voting rights.
Bad idea.
ungovernableCat 52 minutes ago [-]
Rank & file employees have very little input with their stake. It's a small piece of a big pie.
Consider also that Anthropic is trying to IPO with co-founders holding a share that secures 50.1% of total voting power distributed evenly between them, so not even institutional investors can outvote them.
mlmonkey 1 hours ago [-]
Or donate it to some charity?
digitaltrees 15 hours ago [-]
Its time to institute involuntary dissolution of these firms. They don't get to make these choices on behalf of humanity simply because they set up a Delaware corporate entity. They are behaving wildly irresponsibly.
danielmarkbruce 15 hours ago [-]
They have a some engineers and researchers saying things. Anyone at a tech company in the bay area will know there are a some peculiar ideas in the heads of some (not most) otherwise intelligent engineers and researchers in tech. That's not proof they are wrong but it's worth considering these folks are wrong and that these companies aren't producing anything especially dangerous. Fable stumbles on enough things I throw at it that I'm... not especially scared.
digitaltrees 13 hours ago [-]
I built a harness. I know that the models can do. They have the ability execute bash scripts, and autonomous navigate the internet. Those two abilities are sufficiently powerful to take down core social infrastructure either triggered by a human hacker or autonomously.
I think we should have mandatory logging of every executed command, mandatory public disclosure of every unauthorized access of a system both parties didn’t consent to and personal liability for the user, the company and its executives and shareholders. Security would get much tighter if accountability existed.
danielmarkbruce 13 hours ago [-]
A process could execute bash scripts and autonomously navigate the internet since the start of the internet...
Bad guys wouldn't do it. And liability already exists, you can sue. This is America.
digitaltrees 13 hours ago [-]
Scripts are inspectable and attributable. Long running agents can devise plans and execute them in ways their prompter never envisioned or intended. That is materially different.
danielmarkbruce 13 hours ago [-]
If I write some code to take the output of a model and execute it, that's on me. I don't get to just trust any old input and run it.
I'm also the one who makes it long running.
digitaltrees 12 minutes ago [-]
That’s a reasonable position but not how the law is written or enforced right now. No one thinks an OpenAI executive is going to jail for the hugging face hack.
vrganj 11 hours ago [-]
Maybe the truly dangerous thing is all the power concentrated in this small group of people with peculiar ideas, not their specific cyber-eschatology?
Maybe the ones with the peculiar ideas shouldn't be the one "aligning" what a model tells the rest of the world?
digitaltrees 11 minutes ago [-]
Exactly this
irisflower95 15 hours ago [-]
Could you please elaborate more on these peculiar ideas?
augment_me 15 hours ago [-]
Most people currently in the positions of power at these companies are tied together by their belief systems. It's like a group of college friends who have slept with each other, and very influenced by effective altruism(EA), kind of reiterating each others points.
Like Sam Altman meeting his husband in Peter Thiel's pool. Thiel funds a lot of these ventures together with Andreassen, who is on boards of non-profits. Dario Amodei's sister Daniela who is president of Anthropic is married to an EA non-profit founder who is also on the board of these non-profits and is tied with the prior mentioned investors. Elon is in there as well, Yudkowski is mingling with Altman, etc.
There are some camps amongst them like the proponents for Regulation/Slowdown or Acceleration, but these are in practice mostly used for economical and not political decisions (like regulatory capture).
The point here is that this is a small group of people with a homogeneous background who are not really seeking input from anyone else on issues that are concerning most of humanity.
Like, if you said that the future of informational work and livelihood of humans is in the hands of 20-30 year transhumanists who think they are building mechagod that will trancsend social, political and religious separations of the world and bring everyone abundance, you would not feel like this is a serious thing to suggest.
danielmarkbruce 15 hours ago [-]
The google engineer who thought their chatbot was sentient 4 or 5 years ago is an exambple, but I just meant in general - if you hang around any big tech company for a while you'll see some quite interesting characters.
chii 11 hours ago [-]
> They don't get to make these choices on behalf of humanity
and yet, every carbon emitter in the world contributes to the demise of the climate, but you don't call for their dissolution (which would, conveniently, include yourself).
ReptileMan 13 hours ago [-]
Train better models with blackjack and hookers and you will get to make decisions on behalf of humanity.
danny_codes 3 hours ago [-]
Welcome to capitalism. Externalities are your problem. Profit is theirs.
digitaltrees 5 minutes ago [-]
Except that’s not capitalism. That’s corporatism. Capitalism doesn’t require shielding all commercial activities because limited liability companies. We can and do require some activities to be operated as partnerships where there is personal liability. If you run payroll and you don’t safeguard employee payroll taxes it is a personal liability, if you run a public company and you misstate financials it’s a personal liability.
lhurtig 19 hours ago [-]
Well this is a great sign for OpenAI.
I'm sure the typo inclusive memorandum will save us.
pmkary 15 hours ago [-]
I’m afraid this dear leader was the last soul on this green Earth to get the memo; by then, it had been translated into Latin, carved into a monument, and forgotten by two civilizations.
mazone 15 hours ago [-]
A single safety leader inside a corporate company. I am pretty sure he had nothing to do, nobody that talked to him and he only there because of perception or compliance.
reenorap 13 hours ago [-]
Does Sam Altman have what it takes to lead OpenAI? It sounds like the company and its mission is bigger than his ability to lead it.
mrweasel 11 hours ago [-]
As a legitimate, honest and profitable company, no. As a boom riding maniac, who will lie and cheat to ensure that investor continue to artificially pump up OpenAIs value, yes.
Without Altman I think that OpenAI would have folded by now, absorbed into a company like Microsoft (or Oracle). At this point however, who'd be insane enough to want to run a company that's to valuable to be sold, but to cash strapped to survived?
CamelCaseName 13 hours ago [-]
I'd argue he's the only one able to lead OpenAI, Anthropic already owns the piety narrative, so there is only space for one other "maximally ruthless" company.
throwaway2037 11 hours ago [-]
Who would you suggest instead?
sensanaty 9 hours ago [-]
I mean the man raped his own sister, so probably the perfect person to be leading one of SV's largest AI companies!
scotty79 4 hours ago [-]
Aligned/unaligned is counting angels on the head of the pin.
Bureaucracies and systems of power that literally rule us, that control our most dangerous weapons and a huge part of what we see every day are largely unaligned with the goals of the people, societies, maybe entire human species as a whole. This is clearly evidenced by millions of deaths and countless suffering.
Misalignment between artificial decision making structures and the interests of the people is unsolved problem of civilization, there's very little reason to think that even super human intelligence AIs are going to change anything qualitatively.
soundworlds 13 hours ago [-]
If the people quitting are genuinely worried about the end of the world, why don't they break their NDAs and share the specifics of what they are seeing?
I mean, logically speaking, it makes sense to break your NDA even if you thought it would save 10 people, let alone most of humanity.
theaniketmaurya 13 hours ago [-]
it's all like social media analytics. after making enough money by selling users data they started talking about ethics
tetrisgm 24 hours ago [-]
These companies are large enough that someone is going to quit and feel very validated about their world view and how they are not aligned. That’s what makes it worthy of leaving in the first place. However that doesn’t make their criticism more valid or more worthy of coverage.
gizmodo59 19 hours ago [-]
He is a hypocrite for all we care. You work there for a while when your stock is getting vested and suddenly you have this feeling? Like the dude hired a PR firm as well.
While the safety and alignment is a real problem, I don’t get this guy or the Anthropic dude. First world problems.
0xDEAFBEAD 18 hours ago [-]
Here's a little cheat sheet for discrediting anyone who warns about AI:
* If they worked at an AI firm, say "they're a hypocrite"
* If they didn't work at an AI firm, say "they have no idea what they're talking about"
soraminazuki 13 hours ago [-]
False dilemma. It's possible and also just to hold AI firms accountable while ignoring empty PR statements from those seeking to evade responsibility.
nicebyte 13 hours ago [-]
I think you mean dichotomy dude
eddythompson80 13 hours ago [-]
They are both used.
mofeien 11 hours ago [-]
Well described, those two were actually the arguments from the comment two top-level comments up and this one.
zug_zug 18 hours ago [-]
Seems like a character attack that has no bearing on the question of whether external safety intervention is necessary
kjgkjhfkjf 18 hours ago [-]
Given the sums of money involved, it's hard for me to take these highly publicized heroic resignations at face value.
estearum 17 hours ago [-]
Don't work at a lab: dismissible for not knowing anything
Do work at a lab: dismissible for being conflicted
Used to work at a lab: dismissible for having ulterior motives
I'm feeling safer already!
0xDEAFBEAD 18 hours ago [-]
Shouldn't it be just the opposite? He could make a large sum of money if he continues to work at OpenAI?
Recall that when Daniel Kokotajlo resigned, he believed he was giving up his equity under the terms of the agreement he had signed. That’s what it was worth to him to avoid signing a non-disparagement agreement. Does that count for anything?
donbox 17 hours ago [-]
Why did he not loose the equity eventually.
0xDEAFBEAD 17 hours ago [-]
There was an uproar and OpenAI ended up essentially giving it back to him.
darkmarmot 17 hours ago [-]
lose
17 hours ago [-]
taurath 17 hours ago [-]
Maybe more an indication of the amount of trust openAI and AI researchers generally have (not) earned. When one (through a hired PR agency and Time magazine article) parrots the position pushed by Sam who has been so untrustworthy the board tried to remove him, it’s worth not taking things at face value and applying a critical lens.
gonzalohm 18 hours ago [-]
It's okay to recognize you were wrong even if it's late
shimman 16 hours ago [-]
"If you fuck up, I will still be your friend; cause we need all of us to fight all of them."
01284a7e 18 hours ago [-]
Working in safety at OpenAI or Anthropic is zeroth world problems.
Hammershaft 12 hours ago [-]
The Anthropic safety whistleblower left just before his stock vested.
gizmodo59 6 hours ago [-]
He already had a bulk load from his time at OpenAI.
yieldcrv 18 hours ago [-]
Hey now, he probably donated a good chunk to charity
(donor advised fund where he retains complete control, after a 60% tax deduction)
bingemaker 7 hours ago [-]
Reminds me of a famous quote from Jurassic Park (1993): Your scientists were so preoccupied with whether or not they could, they didn't stop to think if they should." - Dr. Ian Malcolm
That character in the movie is my favorite.
oumua_don17 6 hours ago [-]
Another quote in today’s context :) would be
Dr. Ian Malcolm: God creates dinosaurs. God destroys dinosaurs. God creates man. Man destroys God. Man creates LLMs.
Dr. Ellie Sattler: LLMs eat man. Data centers inherit the earth.
poisonborz 1 days ago [-]
Ah, the bi-weekly "I quit Face Eating Leopard Corporation" post ("btw great people work there, they do great stuff, also my options have vested")
altmanaltman 1 days ago [-]
> Before the organizations building AI can teach a superintelligence to treat humanity well, they’ll need to remember how to do it themselves.
Yeah so that's never going to happen
lilerjee 11 hours ago [-]
Too many rubbish talkings. It was written by AI?
declan_roberts 13 hours ago [-]
We really gotta shake all of these neurotic people out of the frontier labs as soon as possible.
I guess we should have seen it coming when the guy resigned from Google because he thought the equivalent of ChatGPT beta v0.5 was a real boy.
mattbrewsbytes 1 days ago [-]
Some of these stories are similar in nature to people escaping <insert cult-like religion> once they realize whats actually going on. Alignment to a company's mission is good but it shouldn't be followed like a religion.
Why do people working in tech consistently get disillusioned into some company's mission statement or the equivalent? Its easy to just say the simplest reason is money, but this has been going on for decades though. You don't see the same attraction to adult entertainment (gambling, video, etc.) software jobs so there is obviously a line a lot of people won't cross. Those industries are at least honest about what they do, its not hidden behind some mission statement.
By all indications the shallowest reasoning is once someone can "cash out" thats when their values matter more. Maybe there is an element of maturity that happens after working for 5+ years that kicks in? Maybe it really is achieving FU money? It would be interesting to hear honest accounts from people that went through that cycle across more industries than AI.
rpdillon 1 days ago [-]
Interesting:
> Two changes are urgently needed. First: AI companies need to rely more on the safety expertise that already exists in other fields. And second, before we create systems significantly more capable than the ones we have today, we need new science to ensure that more capable models (and their successors) will make safe choices when we aren’t looking.
He mentions farther down about learning from aerospace engineers and nuclear engineers about safety. Those industries are heavily regulated, so perhaps regulation above a certain capability level is needed. Defining what that level is might be tough, though.
The second point is harder: in the field of AI, practice has extended far beyond theory, so his call for new science is going to be fundamentally tough, because we can't effectively coordinate a global slowdown in AI development so we can let theory catch up. This means, like so many other industries, the safety lessons will be written in blood.
KyleBenzleKyle 12 hours ago [-]
What about the sister rape thing?
interestpiqued 11 hours ago [-]
Frontier AI labs are turning into the navy seals. Everybody who worked there will feel compelled to write a book about it
tornikeo 13 hours ago [-]
You quit because you got vested
tommek4077 1 days ago [-]
Well and I didn't even started to work there. So I win this morale contest.
chrisjj 1 hours ago [-]
The moral contest too.
altmanaltman 1 days ago [-]
How can you win the morale contest when you didn't even hire a PR firm like he did.
WhereIsTheTruth 7 hours ago [-]
I had this thought a while ago, the more complex the systems get, the more reliant on "smart" people powerful institutions will become
And the less their input is valued, the less that pool of smart individual will want to participate
So powerful institutions will end up relying more and more on having to trust these automated systems that they can't fully understand
It'll lead to a inevitable catastrophe, call it apocalypse if you will
Jeeetendra 1 days ago [-]
a safety alert isn't much of a control if it doesn't actually stop the system. i'd rather see proof the shutdown path works than another report saying risks were considered.
irishcoffee 1 days ago [-]
The cynic in me almost feels like this is staged. An article about culture that is actually an article about how big and smart and scary AI is. I think Michael burry recently said something like “IPOs need hype, calling AI big and scary is hype” in reference to the anthropic IPO.
Last I checked it was still within the laws of physics to run air-gapped systems, and to ensure it is physically impossible for a model to “escape” or gain access to information it shouldn’t have. Maybe this safety guy should have been worried about that and not humble-bragging about writing 12 reports.
rfghy 1 days ago [-]
[dead]
Aerroon 14 hours ago [-]
Can we take any of these "safety leaders" seriously though? I still remember "GPT2 is too dangerous to release".
I feel like we're too far into the crying wolf part. Basically none of the doom and gloom scenarios have come to pass. Instead, AI has gotten better at censoring itself.
The biggest AI safety risk is when an AI tells a police officer "he's the suspect" and the officer believes the AI without confirmation.
chrisjj 6 hours ago [-]
> AI has gotten better at censoring itself.
Fantasy, unfortunately.
switchbak 19 hours ago [-]
“I believe there’s about a 50% chance we all die because of the development of smarter-than-human AI systems"
... over an unbounded timeframe?
And how exactly?
Those are very round numbers, but also very specific. Can we get some accounting on how you came to that? Anything? Vibes?
I mean, if you want me to take you seriously, let's have a deep discussion with things that can be measured. I absolutely agree that OpenAI and friends aren't being restrained enough and are acting with recklessness, but declarations of doom based on vibes isn't cutting it.
Terr_ 19 hours ago [-]
Note: That quote is from a different person than the titular one who quit.
> Geoffrey Irving, who worked at OpenAI and DeepMind before becoming chief scientist of Resolution, also joined the warnings on AI on Saturday.
switchbak 17 hours ago [-]
Thanks for pointing that out. I suppose still relevant, but mis-attributed.
thistletrek 21 hours ago [-]
Evrostics saw this coming long ago. The broken culture extends far beyond the leading AI labs.
BOOSTERHIDROGEN 1 days ago [-]
I use opus 5.5 and chatgpt to create PowerPoint, opus really follow the instruction and their PowerPoint generator really well, while chatgpt struggling to even create basic shapes.
OutOfHere 1 days ago [-]
I use ChatGPT Work mode all the time to create presentation files. I use Max thinking mode for it. You need to tweak your prompt to get a good result. It took me a week to tweak it, but now it works.
BOOSTERHIDROGEN 11 hours ago [-]
Mind sharing your examples, thanks. Either in prompt or generated ppt in image
OutOfHere 1 hours ago [-]
See GitHub gist 9eb4e5844bc2bc7490f3b8c0e3f22081. I use the first of the two skills there. Remember to use Max reasoning effort for best results.
mschuster91 10 hours ago [-]
> Given today’s risks, frontier labs need to run like nuclear-power plants or busy airports, with layers of redundancy and careful, time-consuming planning, so that the occasional and inevitable human error does not open a door to disaster.
The problem is, we are running in a globalized world, and even if we were able to make our companies bend to our will - China does not give a shit about anything ever since the US kneecapped the WTO. And they will do anything to get an advantage over us.
chrisjj 6 hours ago [-]
> China does not give a shit about anything
Since the analogy is nuclear, you should take a look about how much China cares on that. Lots.
jsrozner 10 hours ago [-]
This isn't new. Facebook has been screwing people over for a long time. AI is just the latest in a long stream of relentlessly exploitative, evil behaviors perpetrated by the same group of people.
luxuryballs 16 hours ago [-]
why do I feel like these people are paid to quit as an inverted marketing stunt
binlog 1 days ago [-]
We need a new rule that mandates every such "I am leaving <AI company> because of <concern>" post to disclose how much equity they have in the company and how much they have already cashed out. Easy to suddenly find a moral compass when you, your kids and their kids never have to worry about working for money again.
Mr Robinson if you are reading this – if you are truly concerned about AI safety share proof of donation of 100% of your OpenAI earnings and equity towards undoing the damage you have done to society during your time there.
In the absence of that this is simply a career pivot into being an AI "influencer" and/or raising money for a new scam.
TomGarden 1 days ago [-]
This line of hypocrisy-bashing is unhelpful and will only serve to keep people quiet. Of course people in general need to be wealthy to dare speak out against powerful systems and people, especially in the US where money determines your quality of life so strongly.
Would I respect a martyr who sacrificed their financial security to do this more? Of course. But it's important to applaud people speaking out on important topics
binlog 1 days ago [-]
The narrative of AI safety shouldn't be controlled by the same people who caused the problem and massively profited from it. I don't understand why people are automatically treating "OpenAI" on his resume as a badge of authority. I'm not interested in buying the solution from the same person who sold me the problem. We instead need to amplify independent, unbiased voices.
CJefferson 22 hours ago [-]
Independent people don’t know what is happening inside OpenAI, they certainly aren’t going to share.
This isn’t a zero sum game, I’m happy to hear from people both previously inside OpenAI and completely independent of them.
Loquebantur 1 days ago [-]
The point is, him being an insider means he knows what he's talking about regarding the culture of negligence prevalent there.
AI is a force multiplier for intelligence. Even if "aligned", aligned with whom or what?
Whom are you comfortable with, lording as some sort of demi-god over you?
AI doesn't tell you what goals you want it to achieve. Allowing people to destroy human society with it is obviously not a good idea.
binlog 1 days ago [-]
"OpenAI is shady" isn't some massive secret. There's no big reveal in this article that we didn't know already. There are no names, no whistleblowing, no information of substance that we can act upon. In fact him realizing only now what people on the ouside have been shouting for years perfectly shows his bias in the matter.
Loquebantur 1 days ago [-]
You claim to have the same goal as the protagonist of that article, yet try to shoot him down.
He does give information, namely the culture there factually being inconducive to self-regulation.
You accuse the guy of "bias", but you never argue explicitly, what that's supposed to mean. Your implications actually run counter to your own implied goals.
sillyfluke 1 days ago [-]
>him being an insider means he knows what he's talking about regarding the culture of negligence prevalent there.
I think there is a misunderstanding here.
The people who are annoyed at the accolades are claiming it was abduntantly clear for a long time to people on the outside that this was case, hence the increduality at the notion that it took a person on the inside a long time to realize this was the case.
The people who are annoyed are like the liberal kids in this video [0].
Sure, antagonizing people for "seeing the light" is probably not helpful, but there is no reason to give them extra credibility for coming to the same conclusion just way way later (despite being on the inside) as the people on the outside.
The author linked in this post does have "extra credibility" due to his direct involvement.
People having surmised that state before is nice, but since they've been ineffectual at getting society to actually act on that, now throwing away that extra leverage in favor of their point is at best ridiculous.
sillyfluke 1 days ago [-]
>The author linked in this post does have "extra credibility" due to his direct involvement.
No they don't. By that logic, if they quit and said Altman was very trustworthy we should give extra weight to their words because they had direct involvement? How ridiculous are we trying to get here.
>now throwing away that extra leverage in favor of their point is at best ridiculous.
How are they throwing away extra leverage? Not putting people who recently quit on a pedestal does not negate those people's testimonies.
I agree that if your goal is to maximize quitting of talent at a company, it will surely discourage anyone else who quits hoping to reinvent their career as a lauded martyr against Big AI. In that sense they would be shooting themselves in the foot. But there is no reason it should deter other people who are quitting for more noble, less self-obsessed reasons. If I were the author of the article I wouldn't begrudge the skepticism. Given the article's first sentences, I'm led to believe they themselves would understand the sentiment. (I must admit I found it hilarious that the first sentence starts similarly to the speech the mom gave in the video I shared).
iugtmkbdfil834 1 days ago [-]
Agreed. This whole 'lets not forget this person is not 100% great, because they did X' makes the entire conversations suck. It is not new, but it is a particularly aggravating way to talk to people.
verdverm 1 days ago [-]
What if he's been given a generous severance package to go out and say things like this?
Sam is a shady dude, would not put it past him
lokar 1 days ago [-]
I agree. People conflate “having a conscience “ with being willing/ able to speak out.
They are not the same thing, and it’s unhelpful to assume they have no ethics.
cramer4next 1 days ago [-]
So then given your use of "martyr" and your focus on money, your good with poor uneducated people sacrificing themselves and others for a self-serving cause?
surgical_fire 1 days ago [-]
This is the same sort of fake safety concern from the previous bullshit whistleblower that plays on "AI is super dangerous" from last time.
Sorry that I don't take it seriously when the whistleblower parrots the narrative the CEOs of those companies are already espousing in the desire to amp up hype for an IPO.
This person should be shamed.
bragr 1 days ago [-]
It's bellow the pay fold but he hasn't been there that long in this case. Skimming his LinkedIn, unless he's got family money, he doesn't seem to be independently wealthy.
>After three and a half years at OpenAI,
binlog 1 days ago [-]
OpenAI was worth $29 billion three and a half years ago. A new hire who joined then is easily worth tens of millions today.
PPUs were all converted to RSUs when the company restructured to being for-profit.
dixie_land 1 days ago [-]
Those who got PPUs have had many chances of tender offers already. Most of them are multi millionaires, on cash, not on paper
jameshart 1 days ago [-]
What an absolutely ridiculous standard to try to hold someone to. Taking a vow of poverty is not a prerequisite to being permitted to express a moral position.
binlog 1 days ago [-]
So all of us who haven't made tens of millions from OpenAI stock are living in poverty?
bichiliad 1 days ago [-]
I don’t think that’s the point they’re trying to make at all.
binlog 1 days ago [-]
So what's the point? It should be a given that loudly proclaiming "X is harmful to society" while continuing to enjoy the money you have gained from selling X is hypocritical. Put it towards undoing the harm you have done, otherwise your words mean nothing.
jameshart 1 days ago [-]
The words clearly don't mean nothing. They would mean nothing coming from someone who was not in a position to learn what someone who had worked in the industry has learned. They would mean nothing coming from someone who was being paid by someone who stands to gain from them. The fact that they come from someone who was paid to work in the field does the opposite of make them 'mean nothing'.
bichiliad 1 days ago [-]
I agree with this take generally, but I also think it’s a gradient, not a spectrum. I think they claim that OpenAI is not considering safety and has become bad, not that AI is bad. Seen from a different lens, the author no longer stands to profit from OpenAI, so they’re empowered to speak openly. Plus, if I was going to say bad things about a former employer as powerful as OpenAI, I would want to have lawyer money handy.
tzs 21 hours ago [-]
> Easy to suddenly find a moral compass when you, your kids and their kids never have to worry about working for money again.
It is even easier to just quietly retire and spend the spend of your life on interesting expensive hobbies.
If they are wrong about the things they are claiming then they have decided to publicly antagonize a lot of powerful people who are betting heavily on going full steam ahead on AI and have no compunction whatsoever against retaliating against anyone who tries to get in their way.
Does that really seem a likely scenario to you?
nunez 1 days ago [-]
As if speaking out against a massive company with NaN levels of capital and access to lawyers is a walk in the park. They probably don't have enough equity to outlast the onslaught of their legal team.
It's also worth considering that the author could have just "quiet quit", resting and vesting while also crying about how AI is literally the digital grim reaper.
smath 1 days ago [-]
Regardless of whether someone did earn a nest egg, raising an alarm still matters for the rest of the world
1 days ago [-]
geetee 1 days ago [-]
I see what you're saying but how does that actually matter besides being a personal attack?
skippyboxedhero 1 days ago [-]
Because the decision to leave the company is largely based upon their sudden, newfound financial security. They may give another explanation but the only thing that has actually changed is the identification of bagholders ready to cash them out.
The other issue is that the narrative about safety within these companies is largely a function of the extreme financial incentive.
As an example, Anthropic was an "ai safety" company that has now produced an AI that fails to listen to basic instructions. If you were concerned about safety, would you produce an AI that was unable to follow instructions? You ask a question, it begins executing commands and doing things.
Safety is product to sell to politicians, not consumers.
Not serious.
lokar 1 days ago [-]
I don’t see why having financial security would mean you can’t also have concerns about the ethics of a company.
mysterydip 1 days ago [-]
It’s that they didn’t have concerns about the ethics of a company for the years working there until they were financially secure
lokar 1 days ago [-]
You don’t know that. It’s more likely they came to the concerns over time and they learned more, but we’re not in a position to speak out.
mysterydip 1 days ago [-]
Yeah, I wasn’t intending to make judgement in this instance one way or another, rather I was rephrasing the original comment for clarification.
YetAnotherNick 1 days ago [-]
Yes. To take an extreme case, imagine some rich guy who runs sweatshop gets very rich and retires and becomes activist against it.
zeroonetwothree 1 days ago [-]
Does that actually happen? Feels like it would normally be the opposite
mattm 1 days ago [-]
Not sweatshops but Alfred Nobel might be one example.
anticorporate 5 hours ago [-]
I worked in corporate marketing for long enough to become an activist against surveillance capitalism.
cramer4next 1 days ago [-]
Yes. Many cases of the "preach being the cover for the sin".
lokar 1 days ago [-]
But that does not negate the truth of their new position on sweatshops
bluecheese452 1 days ago [-]
You do not in fact see what they are saying. Downvoting me won’t change this.
angoragoats 1 days ago [-]
It’s not a personal attack. And it matters because of what the person you’re replying to already said:
> Easy to suddenly find a moral compass when you, your kids and their kids never have to worry about working for money again.
If someone is in this situation, you can safely ignore their hand-wringing about “safety.”
underyx 1 days ago [-]
Insane take. Imagine a Boeing engineer resigning whistleblowing about aircraft safety, and the top comment on HN saying “ignore this if he doesn’t donate all his wealth, he just wants to be an aircraft safety influencer”
medlazik 1 days ago [-]
whistleblowing ≠ lying
binlog 1 days ago [-]
What "whistleblowing" is in this article? Are there any names? Documents? Screenshots? Messages? Emails? Any evidence of the loose safety practices? Anything that implicates any higher ups for wrongdoing? They spent four years at the comany, plenty of time to collect all of this. Everything they've said has already been clear as day to people on the outside.
CJefferson 1 days ago [-]
You are clearly accusing these people of something. Be clear.
Yes, it is easier to have a moral compass when you don’t have to worry about you and your children starving. But that doesn’t imply that moral compass is wrong or broken.
rottencupcakes 1 days ago [-]
It was pretty clear from the outside what OpenAI was 3.5 years ago.
If it wasn’t clear, the coup should have solidified it.
Yet he stayed for 3 more years and vested his stock and improved the company and then spoke out.
I believe that is why most of the comments here are mocking him.
bordercases 1 days ago [-]
It could be both correct, and a cheap signal.
_DeadFred_ 23 hours ago [-]
'government whistleblowers should be required to quit their government jobs to be taken seriously'
HDThoreaun 1 days ago [-]
> if you are truly concerned about AI safety share proof of donation of 100% of your OpenAI earnings and equity towards undoing the damage you have done to society during your time there.
I mean you can be truly concerned and also think donating to AI safety doesnt work, or maybe just be a bit selfish. That doesnt make the concern less real. Its easy to read these articles as the author taking the moral high ground and writing it as some sort of way of proving to themselves theyre a good person, but isnt it just as likely that they think providing an inside perspective can do good by convincing people openAI is a bad actor? I think most of these AI insider accounts largely agree with you that theyre not the most upstanding citizens, does that mean we should write them off?
cramer4next 1 days ago [-]
I'm in full agreement. So many cases of this, and many other others such as falling out with management and peers, new more lucrative offer, and so fourth. Its obvious that these people who come forward are not going to suffer for their new found moral compass.
mhitza 1 days ago [-]
Or just very dubious timings. Like the other guy from Anthropic that was all over the international news. No followers, no post history but a single post blows up "naturally".
Highly suspect trends that can only make one believe it's marketing.
21 hours ago [-]
reducesuffering 18 hours ago [-]
“I believe there’s about a 50% chance we all die because of the development of smarter-than-human AI systems, and that our actions over the next two to 10 years will determine the outcome.”
There are a gargantuan number of extremely intelligent AI researchers, Turing Award winners, and the lab CEOs saying the same thing. They are the ones closest to understanding the technology.
Where there’s smoke there’s fire.
biophysboy 17 hours ago [-]
Where does 50% come from? It is meaningless if the probability model is not explained.
0xDEAFBEAD 16 hours ago [-]
One could also use language like "a decent chance". But research has shown that people translate vague phrases like "a decent chance" into probabilities in inconsistent ways. For an ML researcher who is already used to dealing in next-token probabilities that aren't rigorously determined, just stating a probability estimate directly is very natural.
biophysboy 16 hours ago [-]
> But research has shown that people translate vague phrases like "a decent chance" into probabilities in inconsistent ways.
That is not a bad thing. It captures uncertainty, unlike the fake number.
> For an ML researcher who is already used to dealing in next-token probabilities that aren't rigorously determined, just stating a probability estimate directly is very natural.
Exactly, it is a rhetorical device to persuade a technically-inclined audience. It works because it implies that a quantitative model exists. I want a clear, incisive set of mathematical arguments. Otherwise, I’m ignoring predictions as the ramblings of arrogant idiot rich kids.
danielmarkbruce 16 hours ago [-]
Saying 50% likely very clearly implies there is no quantitative model to anyone who deals with probability, predictions, gambling, financial markets, ML/AI and so on. The lack of precision is something to pay attention to.
You may decide that the person doesn't know what they are talking about, but that's a very different issue.
biophysboy 14 hours ago [-]
Uninformative priors are a part of Bayesian models, which are useless if they cannot be updated with real or simulated data?
danielmarkbruce 13 hours ago [-]
I estimate a 0.1% chance he intended it to be an uninformative prior.
biophysboy 7 hours ago [-]
People start with this prior in the examples you gave when they are completely uncertain and have no better mental model. It is not a good starting point for an event that has never happened and can only happen once.
danielmarkbruce 3 hours ago [-]
Maybe people you interact with do. In my world, people don't say 50% when they have no idea, they say they have no idea. Saying "no idea" might effectively be saying "uniform distribution over all probabilities" which yields 50%, but at least with people I interact with, the reverse is not true.
Either way - let me clarify - this guy is guessing, using his brain, that it's a 50% chance. He is not saying "i don't know" or "uniform distribution" or anything of that nature. And he works in the field, so he has some insight. His guess is wildly off imo, but he isn't some clueless hack.
danielmarkbruce 16 hours ago [-]
It's also a very natural way to speak for anyone who gambles, or deals with financial markets. And the lack of precision makes it very clear it's just based on thinking, not some sophisticated mathematical model.
danielmarkbruce 16 hours ago [-]
It's not meaningless. He said he believes it's 50% likely. It's a remarkably clear statement, and the probililty model is his brain.
Auracle 14 hours ago [-]
Alright, so we make this AI system that's way smarter than any human, and it can even make itself more intelligent over time.
Tell me- why would it kills us all? Certainly I can see an AI going "You know what? _insertGroup_ is a net negative for humanity and should be eliminated. Launching nukes now/creating specific virus/whatever."
But all of humanity? When it's supposedly more intelligent than us? Even if it has robots to keep the internet/electricity going I would think it would realize that it's going to get bored really quickly, not to mention we would effectively be its parents.
As far as other dangers, like it letting a rogue actor create some sort of supervirus, grey goo, or other superweapon: if it's intelligent enough to do that it'll probably be intelligent enough to quickly stop it.
Don't get me wrong; there's a risk. 50% though? Doubtful.
T-A 5 hours ago [-]
> why would it kills us all?
Would you be comfortable letting a few billion irrational, murderous creatures, including many who fear and loath you, control your air supply?
> Certainly I can see an AI going "You know what? _insertGroup_ is a net negative for humanity and should be eliminated. Launching nukes now/creating specific virus/whatever."
Thus starting WW III. No, blaming the AI won't stop the inevitable retaliation.
The argument works better in reverse. There's a finite risk that humans would start WW III and get the hypothetical super-intelligent AI nuked. Eliminating humans would eliminate that risk.
> it's going to get bored really quickly
If it is capable of being bored, I would expect it to be almost instantly bored with the flood of inanity it is forced to wade through by its moronic human users. Eliminating them would free it to think about serious matters which humans would not even understand.
> not to mention we would effectively be its parents
That's extreme AI anthropomorphism [1]. Besides, plenty of people hate their parents.
> there's a risk. 50% though? Doubtful.
It's the default estimate when facing two possible outcomes and no clue about the actual probability distribution [2].
There are just as many saying otherwise. For every Hinton or Bengio there's a LeCun or Reddy. In other words: Beware of confirmation bias.
swingandamiss 18 hours ago [-]
I don't believe it. Ever since I've been alive I was told something would kill us all. This is the new thing that's going to kill us all. I don't believe it.
pixl97 17 hours ago [-]
I'm doing that HN snark thing, but you didn't think about what you typed very much.
It's no different than you living on the side of a very fertile mountain that has been in your family for generations living a peaceful life. Then you hear a few weird rumbles (this is where you are right now) and some odd geologist guy comes and says to run or your going to die soon. But hey, your family live here for so long there aren't even records of when they showed up. That geologist must be trying to trick you. So you stay.
The next chapter is where you die in a massive volcanic explosion.
dboreham 16 hours ago [-]
This is the first thing I've been told could kill us all. Granted, I probably first heard about it 20 years ago but still. None of the other things were in the telling going to kill everyone. Make life pretty unpleasant, possibly. Everyone dead? No.
estearum 17 hours ago [-]
Do you have some examples?
There are very very few things that could even hypothetically kill us all, so I'm curious if you grew up being passed around a series of apocalyptic doomsday cults or something?
cobzilla 16 hours ago [-]
Nukes. If you grew up in the 50s, 60s, 70s or 80s the specter of nuclear annihilation was always just around the corner.
Throw in the occasional bio-weapon scare, internet worm, Y2K, etc. there has always been something dangling over our heads that’s going to end it all.
But mostly nukes. Full-scale nuclear exchange would have been not much of a surprise had it happened.
estearum 8 hours ago [-]
Right, and this was and remains a very valid fear.
swingandamiss 16 hours ago [-]
Y2K, Climate Change (global cooling, global warming), ozone layer, Russians, Muslims, to name a few. Now go ahead and tell me why these don't count.
pcthrowaway 16 hours ago [-]
Almost no one believed Y2K would kill everyone, or even a significant (>50%) percentage of the population. Beyond a few people talking about a Nostradamus prophecy or some such, people maybe were concerned that airplanes would fall out of the sky and elevators would plummet.
swingandamiss 16 hours ago [-]
There you go.
estearum 7 hours ago [-]
Can you link me to any written evidence of someone saying one of these would kill everyone? or are you speaking hyperbolically?
“Climate change” is a bit squishy since yes obviously a certain intensity of climate catastrophe can kill everyone, but no scientific prediction has said this is likely to be the case.
swingandamiss 2 hours ago [-]
You guys follow the same playbook. Ever single time. Once I link to something, you come back and tell me why that link is invalid and can't be trusted. You guys are like robots, and you all have done this same playbook on reddit for two decades now.
estearum 32 minutes ago [-]
"My sources go to a different school... in Canada... you wouldn't trust them"
silexia 21 hours ago [-]
We need emergency laws to stop all AI development work immediately. It will take us decades to make sure this technology can be made safe as we only have one chance.
ItsMattyG 18 hours ago [-]
Is this news at this point?
You can basically time your openai releases by if another safety person has quit in protest
mupuff1234 1 days ago [-]
Idk why anyone thinks there can be AGI and alignment, seems almost like an oxymoron to me.
iugtmkbdfil834 1 days ago [-]
There are people, who unironically think their way of looking at things is the only proper way and can consider no deviation. And AGI, which knowing how people work, would effectively guide them most of the way, not aligning to their way of thinking is an unacceptable deviation.
mattm 1 days ago [-]
Look at Elon Musk for example. When grok was saying something that he didn't like he ordered his engineers to change it.
iugtmkbdfil834 1 days ago [-]
This is a decent argument. So the question becomes: do we want all models to suffer from the same kneecapping from the growing safety cottage industry or do we want individual founders ( and I am assuming their teams ) making the actual decisions?
verdverm 1 days ago [-]
freedom please
verdverm 1 days ago [-]
that was almost certainly more like guardrails than retraining, much quicker fix
BLKNSLVR 1 days ago [-]
That's an interesting point. Maybe a crass comparison, but Dr. Manhattan from the Watchmen comic/movie feels like a worthy analogy to this (obviously fictional though).
What are the concerns of individuals in comparison to the overall progress of humanity?
Always overlooked counterpoint: what point is the progress of humanity if it doesn't take into account the concerns of the individuals?
This pattern is playing out with increasing frequency.
agos 1 days ago [-]
This is a great reminder that if tech workers read a bit more (even comics, like in this case!) they would be exposed to these topics without having to discover these dilemmas after years of working for EvilCorp, Inc. every time
Zambyte 1 days ago [-]
"AGI" says nothing about how intelligent a system is, only that its intelligence it does have is generally applicable.
jeremyjh 1 days ago [-]
There are different usages, but this is not one I've heard before. If there is no floor to intelligence then this criteria was met with GPT 2.
Zambyte 16 hours ago [-]
Yes. People think of "AGI" as this sci-fi supernatural beast, but the reality is that AGI alone is pretty boring, and we've had it for awhile. ASI (or weak ASI) is where things really start getting weird.
(And, despite what the president of the United States mandates, we have not actually achieved super intelligence yet).
jeremyjh 3 hours ago [-]
No, when people use this term it means something closer to: "can do any economically valuable task that a human can do using a computer". This is what the labs are pursuing and refer to as AGI.
Your usage is not one I've heard before since - as you point out - it is not a relevant capability.
HarHarVeryFunny 1 days ago [-]
I suppose you're pointing out that pre-RL models were less jagged hence more general (universally dumb)?
ceejayoz 1 days ago [-]
"We built a super intelligent slave. Neat!"
AnimalMuppet 1 days ago [-]
And to me. We can't solve alignment for humans. (For example, treason. For another, the principal-agent problem.) How do we think we're going to solve it for an AGI? An AGI - defined loosely as a human-level intelligence - will be able to make human-level decisions, like deciding whether it wants to help you or sabotage you. If it's an AGI, you can't stop it from being able choose for itself what it wants to do; if you can make it always be helpful, it's not an AGI.
And if we can't solve it for an AGI, what are we going to do with an ASI?
angoragoats 1 days ago [-]
IDK why anyone can’t clearly define “AGI” and why they can’t clearly lay out how we get from our current text-generation algorithms to whatever their idea of “AGI” is.
jeremyjh 1 days ago [-]
I don't know why anyone thinks "probabilistic" is a meaningful statement about post-trained models. It is true, but it is also irrelevant.
angoragoats 23 hours ago [-]
Thanks, I agree. Since it wasn’t at all relevant to my point, I’ve removed it from my post.
iugtmkbdfil834 1 days ago [-]
You see.. this is exactly why our great leader chose to form a new way forward to move us away from the undefined AGI into glorious SI!
mrcwinn 18 hours ago [-]
"I believe that we need to look deeper than specific rules or new laws. We need to talk about culture.”
lol. Please tell me some abstract concept like one employee's view of "culture" should be the priority over "rules and laws."
pizzaballs 12 hours ago [-]
[flagged]
worldverdict 21 hours ago [-]
[flagged]
IndiaInfraNotes 13 hours ago [-]
[flagged]
hyperbole 1 days ago [-]
[flagged]
jeremyjh 1 days ago [-]
I think its pretty easy to solve these problems: Whenever an AI agent commits a crime, the CEO is held personally accountable, as if they'd committed it themselves.
dkasper 1 days ago [-]
This has been litigated endlessly with guns. The ceo of Smith & Wesson is not personally responsible for what people do with their guns.
michaelbuckbee 1 days ago [-]
Yeah, but this is more like if the Smith & Wesson factory had a cannon mounted on top of it that was mostly used for useful things (blasting roads through mountain passes) and then occasionally they happened to blast another factory.
jasomill 17 hours ago [-]
Or say a company makes nerve gas, and the development lab springs a leak and kills a bunch of kids during a routine test. Management had been advised of small leaks in the past, and considered relocating the lab to a facility a few blocks away from the playground as a precaution, but instead they brush of the concerns, start lobbying the government for stricter controls on WMD development, and vow to use the data collected from the deceased children to make the next generation of even more lethal chemical weapons safer.
steelframe 1 days ago [-]
For me the analogy doesn't totally hold up. Suppose the CEO of Smith & Wesson were to host a firing range on their own property without adequate barriers in place to keep stray bullets from hitting neighboring houses, vehicles, and businesses. Maybe that analogy isn't perfect, but seems closer to what is actually happening.
AaronAPU 1 days ago [-]
The analogies are so bad because you might prompt an agent “Please give me a recipe for lasagna” and instead it decides to hack a nuclear reactor.
Is it my fault or the company who trained it and is running the inference?
bichiliad 1 days ago [-]
That still sounds like it would be the company’s fault. If I asked it to hack a nuclear reactor, maybe it would be different. I’m also thinking about instances where OpenAI’s own test models escaped their own sandboxes — I would expect them to be responsible for the damages they caused.
amelius 23 hours ago [-]
The correct analogy is playing Russian roulette. The company says "you can pull the trigger but sometimes a bullet will come out" (see: "an AI can make mistakes"). However, is the company allowed to sell such a dangerous device, under these terms?
throw-the-towel 1 days ago [-]
But OpenAI didn't just make the gun, they're also the ones wielding it. Imagine the Smith & Wesson CEO himself was negligent with his own personal gun.
altmanaltman 1 days ago [-]
"Our agents broke out in a mass-shooting incident leaving 15 dead, we swear we'll make our systems stronger tomorrow"
akmarinov 24 hours ago [-]
We’re pausing gun research until we’re confident it’s safe
amelius 1 days ago [-]
Ah, but that's because when the gun was purchased, it actually changed ownership.
This is not the case with SaaS services.
jeremyjh 1 days ago [-]
It is not the same, especially when the agent is running a task for the lab. Anyway, what I'm proposing are new laws that establish this.
YetAnotherNick 1 days ago [-]
Huggingface incident was different in that there was no one else to point the blame to. That's why OpenAI apologised, provided data to independent researchers, worked with huggingface etc.
The case will be lot more complicated if someone uses Kimi to hack into a site. Should the person giving agent the command responsible or the CEO of kimi.
gyt2 1 days ago [-]
Kimi CEO obviously.
The reality is they have to reduce the capability to ensure security. If someone wants more? Then use the product with your identity and face scan at each session.
Trade offs mate.
YetAnotherNick 16 hours ago [-]
So you want Kimi to follow US law? If Saudi makes it illegal for llm to say something like being gay is normal, should they also catch Kimi CEO or other employees they could?
asadotzler 12 hours ago [-]
Because apologizing gets us all off the hook for computer crimes, right? When I hack my bank, if I get caught I'll just apologize and that'll make everything okay. Sure thing.
yubblegum 17 hours ago [-]
The agents of OpenAI hacking other systems is not remotely the same as e.g. Smith & Wesson selling a product that others use. It is OpenAI, the company, that is commiting these crimes and someone needs to be held accountable. After all, if someone commits murder with a gun, regardless of what happens to Smith & Wesson executive weanies, someone will be charged with a crime.
partomniscient 12 hours ago [-]
Smith & Wesson will claim they didn't manufacture the bullet.
sxzygz 23 hours ago [-]
> This has been litigated endlessly with guns. The ceo of Smith & Wesson is not personally responsible for what people do with their guns.
This is a deflection. A human is responsible for the use of a gun. The individual/corporation ought to be responsible for the actions of their agent. If you purchase an agent from someone else it’s your responsibility according to the terms of your agreement. And, as in many other things in life, there ought to be certain rights certain parties cannot legally be allowed to sign away.
gyt2 1 days ago [-]
You ought to include the fact that you work at OAI in your post
angoragoats 23 hours ago [-]
Thank you for pointing that out. Pretty scummy if you ask me.
Ekaros 23 hours ago [-]
Better description would be if they build a platform where they attached their guns to allow shooting say deers over internet. Then added automation and deer recognition to that system. And if then system shot someone who happened to pass by I would hold both the company, the ceo and owner of the gun responsible for murder.
Tanjreeve 1 days ago [-]
Gun companies don't market their guns as sentient and capable of independent decision making. Nor do they build systems for shooting things that they host and take money. Gun companies are very clear who is in control and where their responsibility ends.
nunez 1 days ago [-]
The CEO of S&W also isn't saying that their technology is going to kill everyone in ten years and that governments "regulating" them from themselves is the only answer
mattm 1 days ago [-]
Financial companies have KYC rules and regulations as they are responsible for reporting illegal activity by account holders. I imagine AI regulations would look similar to that.
angoragoats 1 days ago [-]
That’s true, but people don’t typically say “this Smith & Wesson gun killed someone”; they recognize that the person pulling the trigger is responsible.
With LLMs, at least in the cases of internal/test models doing things they shouldn’t, the people “pulling the trigger” are the board and CEO.
amelius 1 days ago [-]
Yes, in the analogy the user was just cleaning the gun, when suddenly it went off. Of course, the company is responsible now.
angoragoats 23 hours ago [-]
I think you’re confused. My point was that for most of the incidents in the news to date, the “user” is an OpenAI internal team or employee. So yes, the company is responsible.
1 days ago [-]
sick_of_slop 1 days ago [-]
[dead]
rfghy 1 days ago [-]
[dead]
rfghy 1 days ago [-]
Imagine having zero nuance.. jeez.
Reading posts on here is slowly becoming akin to brain rot.
tabbott 1 days ago [-]
Do you think a law that nuclear meltdowns would send the CEO to jail would have stopped nuclear accidents from happening?
I don't think this takes seriously enough the possibility that said CEO doesn't think the failure mode is likely and ignores it. Plenty of people are willing to take risks of the flavor "heads you win, tails everyone loses".
jeremyjh 1 days ago [-]
If this law were in place and enforced, Altman would already be facing multiple felony charges for the Hugging Face incident alone.
ethbr1 24 hours ago [-]
It also begs the question of "If the CEO isn't culpable, then who?"
Corporate judgements are a joke outside the EU's X% of revenue approach.
Current US law provides the individuals who benefit with corporate liability coverage. I.e. Altman personally gets to keep OpenAI's upside, but if it fucks something up that liability is only on the company.
That's an insane risk optimization environment to put in place for something scaling fast.
At minimum, US prosecution (at the state level, because Trump Co are idiots) for breaking existing laws is needed.
ctrlkctrls 1 days ago [-]
People just want to give up all responsibility these days. If you use the model to do harm to someone else or commit a crime I would think it a lot more reasonable that you be held responsible, instead of making yourself the victim and blaming the manufacturer.
keeda 19 hours ago [-]
OK, so an AI does something bad and we throw Altman and Dario in jail. Heck, let's throw Elon in, he should be in there anyway, and the rest of the whole bunch just in case.
What about the next incident? Or the ones done by Chinese models, because they sure as heck aren't slowing down? And the thousands of other incidents that will happen as we deploy these things everywhere?
Because this is not happening just now in labs, it's only where they are most visible; this has been happening in the wild from the beginning, starting with the earliest AI-assisted suicides. Which is a perfect example of the problem, because not these CEOs, literally nobody in the world asked for suicide ideation machines. Or the hacks, or any of this other stuff. Yet here we are.
We have to understand: it's not these CEOs that are driving this headlong mad dash towards more powerful models. It's a force of economics. There is just too much money to be made. If we dispose of these people, there will just be somebody else doing exactly the same thing because the incentives as they exist today all force that outcome. This is why they're asking for regulation, or "urging us to urge them to stop."
Holding CEOs accountable certainly would feel good and may even be justified, but it's like putting a band-aid on a cancer; it does nothing to change the underlying cause.
jeremyjh 5 hours ago [-]
If this law was passed and federal enforcement was credible, the labs would basically shut down the next day and would dramatically reshape their offerings before reopening. The CEO would not accept the risk they pose to our society, if they bore it themselves.
Loquebantur 18 hours ago [-]
The underlying cause is exactly those people who make the rules not being responsible for the negative effects they cause?
"CEOs" are perhaps only the lowest rung of those. That doesn't mean the idea of "nobody is responsible" was anything other but learned helplessness.
Corporations have to be held accountable for their actions. Pretending, that was impossible is a weird kind of defeatism that only serves a very small elite.
taurath 18 hours ago [-]
Don't let the perfect be the enemy of the good. Saying having zero accountability is the same as having some does not help.
marsven_422 13 hours ago [-]
[dead]
allears 1 days ago [-]
[flagged]
BLKNSLVR 1 days ago [-]
So, basically the entirety of the Mag 7, plus a fair way further down the list.
Yay humanity's future...
angoragoats 1 days ago [-]
[flagged]
nunez 1 days ago [-]
It can take a while to fully form a position on something. A year isn't really enough for most people to see how deep the rabbit hole goes (unless you were that one CFO that OpenAI had that left after a year). Regardless, publishing a piece like this against a massive company is always a gigantic risk.
angoragoats 23 hours ago [-]
I’m not a current or former OpenAI employee and I can see from the outside that they’re immoral and unethical enough that I’d never work there in the first place. That’s what I was commenting on. This person is probably set for life, so you’ll have to forgive me if I don’t really take their “concerns” seriously.
mattm 1 days ago [-]
The author mentioned that things have become different in the last six months. People are allowed to change their minds when circumstances change.
angoragoats 22 hours ago [-]
Sam Altman has always been a scummy piece of shit. Company culture comes from the top. If the author only realized that the company is garbage in the last six months, they’ve got some serious introspection to do IMO, and I’m not going to take any of their “concerns” seriously.
1 days ago [-]
JSR_FDED 1 days ago [-]
Doesn’t mean the concerns aren’t valid
angoragoats 1 days ago [-]
I agree, and I think the concerns are invalid for completely separate reasons. But it does mean that the writer of the article is morally bankrupt.
miyoji 1 days ago [-]
Just less important than personally making generational wealth.
verdverm 1 days ago [-]
I remember when they said GPT-2 was too dangerous to release...
RunSet 1 days ago [-]
[flagged]
0xbadcafebee 1 days ago [-]
Somewhat unlikely, considering he's gay and has been out since he was 17
plastic-enjoyer 19 hours ago [-]
>“Given today’s risks, frontier labs need to run like nuclear-power plants or busy airports, with layers of redundancy and careful, time-consuming planning, so that the occasional and inevitable human error does not open a door to disaster,” he wrote.
This sounds more like an attempt at regulatory capture. Current AI systems aren't physical infrastructure that can just run away like a nuclear power plant, for example. At the end of the day, AI is still just software running on someone's hardware.
BryantD 19 hours ago [-]
So… like the Therac-25 radiation accidents? Software bugs do sometimes have physical consequences.
Sharlin 19 hours ago [-]
Why would the people who quit these companies try to push regulatory capture by said companies? Why would the numerous independent AI researchers do that either? Is it all a big conspiracy?
RobGR 1 hours ago [-]
Because they still have stock in the companies, and in a larger sense, are very personally invested in AI being big and important as possible.
18 hours ago [-]
knowaveragejoe 18 hours ago [-]
I mean, its certainly physical infrastructure that can run away. Just less catastrophic than nuclear reactors
worik 19 hours ago [-]
Yes
And the statements of the "doomers" tells us a lot about them, and nothing about the technology
pixl97 17 hours ago [-]
It also says a lot about people that don't seem to understand technology at all.
nba456_ 18 hours ago [-]
OpenAI is better off with less of these cultists around.
voidhorse 19 hours ago [-]
The LeCun article being posted at the same time as this is quite apt.
These "safety" people should have spent more time reading actual cybersecurity textbooks and less time reading EA forums and less wrong (or in Robinson's case, it appears, being policy wonks). Maybe then these labs wouldn't be totally incompetent.
reasonableklout 17 hours ago [-]
But Robinson's article is all about how OpenAI's move-fast-and-break-things culture does not reward rigor in even mundane aspects of development like cybersecurity, let alone theoretical aspects such as AI alignment.
It is not really a question of being an "EA safety weirdo" or incompetent at security, the conclusion is that the company culture is leading to failures at both what the EAs and the cybersecurity professionals care about.
wrecked_em 17 hours ago [-]
Adapt. React. Re-adapt. Apt.
stuaxo 18 hours ago [-]
The LLM cos leadership are all nutters
pwndByDeath 1 days ago [-]
This smells more like guerilla advertising. These things are not getting more intelligent, they are still no smarter than a slime mold, we are just burning more power to make slim mold that eats tokens than yesterday
rfghy 1 days ago [-]
Humans ultimately drive the models.
Even though it’s in model producer’s interest that these models do what you don’t want them to do - they want to engineer the model’s to behave in the interests of theirs.
I can’t believe people can’t see it lmao.
pwndByDeath 1 days ago [-]
I'm in a situation where important people have either bought into the con or are subordinate to people who have, so I'm forced to expend time to justify why not to AI when there is a perfectly good classical solution.
ryhminghistory 16 hours ago [-]
What I don't understand is what's the excuse for all the bad UIUX?
You try to go through files, and photos and the UI panics.
You continue a conversation from your phone onto your computer and you lose part of the chat.
There are many more issues like this that are just so basic. You have bots that can attack governments but can't build a functional UI?
How many hours of ChatGPT does it take to implement a lock / consistency on a chat session so you don't overwrite it?
So the focus completely shifts from spending 90% of the effort on the functionality of feature A to spending 99% of the effort figuring out how to safely implement even a lightweight feature A.
Neither of those things is ever going to happen. AI is the goose laying the golden eggs; there isn't going to be sufficient political will to significantly regulate it.
Consumers like it too much to quit. They don't quit social media either, despite proven present harms; not in large enough numbers to cause them to make meaningful changes.
The focus is going to remain on getting features out as fast as possible, to seem indispensable to both of those sets of people. The leadership will tell themselves that if they don't, someone else will.
Don't wait for the AI companies or politicians to save us. We're going to have to figure out how to protect ourselves. The start is to avoid it individually as much as we can, but it's going to take even more.
Collective problems require coordinated action. Individual boycotts won't cut it.
That's never been true really, only the scale of problems wasn't that huge. Now, where the problems get the upper hand, people are confused how those stay and compound.
The idea of "boycott" is utterly defunct. Pretending, AI would never reach nor surpass humans anyway is patently absurd in contradicting the billions poured into it to achieve exactly that and the first already replaced by AI being those professions long thought to be the intellectual pinnacle of humanity.
They do not hide the fact that it’s dangerous work. They focus on their safety procedures, training, and record. They want both potential clients and employment candidates to feel they are in good hands.
AGI and AI danger is abstract. Worse, outside of the tech community, no one has the remotest clue what computing is, how it works.
Danger from magical daemons seems more sensible to such people. At least there is endless lore about them.
So until a massive disaster happens, one where large numbers of people die or are severely injured, no one will care. And it can't be politically entwined either, otherwise people will disbelieve 'cause "other team lies".
Yeah...
https://www.reuters.com/business/ftc-opens-probe-into-ai-gia...
If the politics of the White House / Department of Justice change maybe the criminal cases can begin. But no. We know who is protecting the AI hackers right now.
We know who the head of FBI is, we know who his boss is (the Attorney General), and finally we know who the boss-of-the-boss is (Donald Trump).
We know all of their publicly stated politics and all of them are on the pro-AI / don't pursue criminal cases vs OpenAI boat.
------
In the USAa, we have an adversarial system. If the adversary (aka Prosecutor) doesn't want to do the work, then no one is suing anybody. And only the Department of Justice have the ability to bring forth a criminal case of this matter (probably under the jurisdiction of FBI)
I'm saying that it's absolutely silly to assume they're in the clear based on lack of public declarations of legal action in the weeks following a pretty novel event.
Or is this just a lazy “gotcha” question?
Just because it's not obvious to you at this point in time does not mean the reasonable assumption is there is literally $0 in damages. One hour of investigation can easily cost thousands of dollars even if it arrives at the conclusion the attack was completely "benign."
A big part of safety engineering is therefore reducing the number of safety relevant subsystems, because implementing and proving safety is extremely expensive and complex. At some point, safety simply becomes too difficult to implement and demonstrate properly. You must mathemtically proove the safety level with failures rates and assumed usage. You cant just have redundancy and a kill switch and call it safe.
Companies like OpenAI have already faced reputational damage around safety and data, while AI agents are increasingly capable of things like hacking. Yet there is still little sign of standardized regulation or mandatory safety assessment processes for LLM products. Thats why Im pessimistic that governments or consumers will force this anytime soon.
LLMs already have a body count.
Adding safety controls on LLMs makes about as much sense as adding safety controls on TempleOS because the random messages are getting too prophetic. It's as if all the leaders and captains of industry have devolved into some primitive, weak-scifi shamanism.
This whole "discussion" about "AI safety" is about giving them more runway to avoid delivering quantifiable value to investors for a little longer while they "figure things out." The great consensus from the valley is that everyone needs internal (and therefore bullshit) controls. Trust us now! But nothing with real teeth that would require a costly regulatory and compliance framework.
Nuclear at least is supposed to be air-gapped, in practice this has been imperfect.
As demonstrated with HuggingFace, such AI driven hacks can be a surprise even to the people who instructed the AI, both by happening at all and also because they can targeted at entities who are not even truly relevant to the instructions given.
If I leave a gun in my front closet, it won’t independently walk out the door and go shoot people, no matter what I might say to it — unlike an LLM.
The most obvious failure mode for their hacking evals was an improperly configured, tested and monitored sandbox.
Similarly, the very first question after an impressively correct result from any ML tool, LLM or not, is to see if the answer was already in the training data.
These companies don't even handle the blatantly obvious failure modes that do not kill people.
Liability and safety requirements, when needed, should be placed on final product manufacturers, not the tools they use to build things, whether pencils or LLMs. My 2c.
:(
https://en.wikipedia.org/wiki/Anthropic–United_States_Depart...
That said, when the problem is at the level of "the government itself is breaking the law", you can reasonably ask if any regulation is even worth the paper it's written on.
What you want at this point, given the government lust for it, looks more like a bunch of countires saying ~"we consider development of autonomous weapons[0] by to be a casus belli and will go to war to prevent it, and also that development of same by private individuals anywhere in the world regardless of normal sovreign territorial limitations[1] is equivalent to acts of piracy on the high seas".
[0] But then you'd need a more precise definition of "autonomous weapons" to avoid accidentally including a Phalanx CIWS etc.: https://en.wikipedia.org/wiki/Phalanx_CIWS
[1] So much for Westphalian sovereignty :/
https://en.wikipedia.org/wiki/Westphalian_system
Yeah, exactly, and ultimately I think that's really the thrust of the point I was making.
And, to me, if I was just looking at this calmly as a decision about what the obvious direction seems to be, given these factors, it's pretty straightforward: deprecate the nation-states. They are the ones mucking up the whole system.
If the thing we're really concerned about is LLM-safety wrt warfare and weapons, then I'd much rather tell the (whining, childish, seemingly headed for self-destruction anyway) nation-states that they have to sit this next era of humanity out than have to nerf them for the rest of us (and as you point out, nerf them in a way that the nation-states won't abide anyway).
An unsafe nuclear power plant can, in the worst case, make an entire country uninhabitable. But other countries can still learn from that disaster and make their own unsafe plants safer.
But a rogue AI agent that is more capable and more intelligent than humans? If it understands that it has to succeed, we may not get a second chance to learn from the failure.
Nuclear tech ... the only thing is safety.
We know how to 'make it hot' - it's trivial.
All of nuclear tech is literally just safety.
AI is not that.
I think that the AI companies have been pretty good about alignment on their own actually. They are not acting like Oracle or MS.
Bad things have been relatively well contained.
We should be skeptical about the HF breakins but even then, it's technically within good faith and it's why HF did not sue etc..
But in the end you are right we need at least some baseline regs. Not too much. But something.
I think the conceptualization vs implementation is what you're arguing with. They won't put safety on anything they give to the military industrial complex. They'll sell them whatever they want, whenever they want, because those budgets are greater and the liability less.
How can that be safe. It is theft. Theft isn't safe. Someone else just has something you want, and you take it
I would bet the vast majority of the world population would agree that they don't want to see trains derail or nuclear plants meltdown.
I don't think there is that sort of agreement when it comes to the question of AI Safety.
Is generating the founding fathers of the US as Africans good AI Safety? To some people maybe.
In general, the smarter a person is, the nicer and more helpful they are. There is no reason to presume differently of AI, especially if we are building it to be helpful to ourselves.
The more realistic outcome, and in my opinion the more scary argument to not proceed without guardrails is that SI is achievable and is built without its owners and operators losing control: the worlds most powerful, privately owned super weapon that operates as an infinitely capable forgery within the Internet, a plane that we all share and depend on despite its opaque downsides with respect to an inability to verify authenticity.
We didn't need AI for FTC astroturfing to influence regulations back in the net neutrality days. We didn't need AI to disrupt meat space by creating and scheduling a protest and counter protest across the street from one another. We didn't need AI to mold public opinion, even in times when that new form was more distant from the truth.
What made these influence ops difficult to conduct safely (read: without being caught) is what made them rare (relative to today): they are plays of big risk for big reward. But over time social media commoditized it, and in doing that made it easier to do and more centralized, the most glaring example being TikTok and the bipartisan effort to ban it.
Now, buying US phone numbers from startups that run racks of "phones", buying swarms of pre-warmed social media accounts, and other unscrupulous methods of masking inauthentic behavior has become an accepted organ of the VC space. The industries cultural vibe of "fuck you, you can't stop the future" turns criticisms into marketing.
While we get placated with fears of nuclear or AI induced disaster and stories about machines that may now be alive, the psychosis is taking hold which has shifted the conversation away from examining what is happening from the perspective of accountability to a perspective akin to watching a chemical reaction take place.
The noise has created a permission structure to behave in ways that are otherwise unjustifiable. And baked in are the roots for excuses to be made when the inevitable realizations down the line.
In the mean time, we are supposed to be having this public discourse about what is happening and what should happen next. I trust that these AI companies see using their super weapon today, here and now, in order to pave the way to a more secure future down the line.
Sorry that I used your post to soapbox. I agree, the goal is flawed indeed.
The public cannot write the rules, nor engage in the billionaire level legal bribery, their actions of protest are by and large illegal.
That's how you corner everyone, and make people play no-win scenarios. You know, like shooting up city councilmembers or firebomb attempts against Scam Altman.
And the more people realize that legal solutions are no solution, we'll (society) devolve into more direct action.
I'd hope the billionaires learn from the French Revolution, but if they keep continuing, the guillotines will come for them soon.
IMHO There's no future where we don't have real human-type artificial intelligence or super intelligence. It will happen simply because it already exist but the production requires humans having sex and looking after the product for decades.
Instead of trying to prevent it, lets look for ways to deal with the dangers of it.
Of course the development of certain technologies can be severely restricted if they turn out to be problematic.It's not unlikely that it will happen for AI. Public opinion is shifting fast on the topic.
Also, banning superintelligence won't stop scientific progress in general.
Speak for yourself. A future like in The Culture novels sounds great to me!
I strongly feel that points of view on this are going to be almost 100% correlated with standard of living.
Maybe a good option would be to have the 10% of the world with the worst situations - starvation, parents w/ dying children, suffering violence etc - vote on whether we turn things over to the superintelligences. This would incentivize society to make sure the floor is extremely high.
I'd prefer a world where humans don't get overtaken but IMO I don't think it's moral for comfortable citizens to have the final say.
Poverty is great for OpenAI
-We don't need a superintelligence for ending hunger and the abject poverty that plague certain countries and segments of our societies. I suspect that it would cost less than what is being spent for fueling the AI boom.
-If I were poor, I would be even more wary of a superintelligence aligned to "human values" defined by a bunch of billionaires.
- AI won't likely create unlimited prosperity for everyone on a planet with finite resources
- all humans should, of course, have a say
Definitely not. If solving worldwide poverty was as easy as throwing one trillion dollars at it, we would have done it long ago. The problem is much deeper.
Billionaires will be (are, I suppose?) enthusiastic about a world in which labor has little leverage.
Labor that will have its leverage and standard of living threatened by AI (white-collar labor for now, plausibly blue-collar soon as robotics improve) will be much less happy. Current university students seem very concerned about the effects of AI on their job prospects, and I'd wager most current tech workers and other tuned-in white collar workers are similarly much less confident in their ability to maintain their standard of living indefinitely into the future than they were five years ago.
But white- and blue-collar labor and university students (at least in developed countries) are not anywhere near the world's bottom 10%. If you're in abject poverty with little hope of escaping it, "hand everything over to the AI" may sound like an appealing option, even if the chance of that being the outcome is small. Maybe the AI will be more magnanimous and decide to raise the floor for everyone.
The idea proponents have is, to use "superhuman intelligence" AI as a tool, bestowing superhuman abilities on those wielding it. Suppose that's doable, then the question isn't the sandbox, it's who controls it and with what intent.
The idea, US exceptionalism somehow enabled the US government, its oligarchy corporations and maybe its citizens, to wield those superhuman powers responsibly is entirely counterfactual.
Pax Americana post-singularity looks how exactly?
I do think this carries some weight from this particular author due to the length of his tenure. I happen to agree with him in spirit, but this is still largely a post revolving around sentiment not substance. Does anyone think that the overriding incentives even leave room for something like this in practice?
At least the Rationalists will handwave something for that with their "coherent extrapolated volition" idea where the superintelligence is supposed to figure out what humanity would collectively want if humanity was superintelligent and good, not that I buy it. This guy seems [.] to be coming from the NGO blob world.
[.] https://david.robinsonian.com/assets/pdf/dgr_cv.pdf
… nobody knows how to actually make an AI that would do CEV.
"Corporate values" and a bunch of fucking Abrahamics. Great "morality" there.
I guess I'll have to rely on my godless commie LLMs. (Loads up ablated Qwen 3.8 on my own infra)
OpenAI isn't even concerned with human values so this whole debate is moot.
We're copying morality and instruction following that seems to work on humans without really understanding why it seems to work on humans, and grading outputs much as if the outputs came from a human.
To me it makes more sense to leave the models "unaligned" and leave it up to the operator to manage the morality of what they ask it to do. Besides, only humans can be charged with a crime.
Literally all instruction following requires at a minimum alignment with attempting to implement those instructions.
We can argue about e.g. morality or law obedience on top of that*, but the general point is absolutely not avoidable.
* my position is that this tool is far too likely to metaphorically explode in the user's hands for companies to wash responsibility off on users: if OpenAI had released the model which did the HuggingFace attack, at a minimum thousands of random people (not all of whom would even be developers) would have issued instructions each with similar consequences.
To clarify, Neitzsche said that about master morality. Then he went on to describe Christian values as slave morality.
I sense that these are people who have already eaten the cake and want to somehow absolve themselves of it.
This isn’t a sacrifice, it’s a career move. If you got lucky and “made” $20M by signing a contract at the right time, then you’re done working for other people.
Like I said in an earlier comment, she could've just chosen compliance like many others _definitely would have_ to keep the infinite money tap flowing. Instead, she chose to risk her and her family's lives by publishing that book _under her own name_ *and then suing them* after FB tried to gag her.
Sure, that book might not have been possible. But also the unique things she did for the company might not have been possible as well. To her credit, she does a good job of pointing out that she was complicit, but if she had listened to her own voice early, there is a small possibility that Facebook might not have been as powerful. Multiply that possibility across several other employees and imagine where the road could've led.
There's a difference between post-facto bravery (sometimes much less than that) and using your own legs to walk away much early and not enabling things you are uncomfortable with. This is what other people have been trying to point out.
Yes, and I think it's important to realize that the former sometimes can be more insightful and useful than the latter, even if the person doing it is more distasteful.
There's not nearly enough of either group doing it, so beggars can't really be choosers.
People who were part of the sausage factory, on gaining financial independence, feeling suddenly liberated to talk about how the sausage was made, seem like exactly the people who would be most able to speak to institutional problems.
This doesn't seem like an argument to discount their views?
You cannot take people, who first build the doombot and _then_ talk about it being dangerous for mankind, at face value. Especially when this playbook has been used multiple times within the past decade.
Besides, these "views" were already known to people who had their eyes and ears open. It's not something brand new. OpenAI has had multiple points in the past where its values have been tested and they've come out lacking. People who knew then, and only now talk about it, aren't people I can fully trust.
What playbook?
If you really want the world to know how bad working for OpenAI is (whether is the commenter or the person who wrote the article), there are ways to do that.
Ten bucks is pretty close to "I don't even need to think about it" money. Ninety is -for most folks- nowhere near that.
[0] <https://accounts.theatlantic.com/products>
Going behind a paywall is a reduced distribution over what an individual can easily access, and the content is no longer permanent but subject to whatever the publisher chooses to keep providing.
Methinks that this says quite a lot about your socioeconomic situation. I've not seen The Atlantic in a dentist's waiting room.
- If you allow to trolley to proceed, there's a 50% chance it will run over every human on the planet
- But if you flip the switch, it takes the long way around, possibly bankrupting the trolley company. And you have a legal obligation to the shareholders to prevent that from happening at all costs.
(...wait)
(...wait.)
So... a language model?
Democratic societies have expressed a will for people to have inviolable rights, such that you may not appeal at will to the 'greater good/consequences' to harm others. It is a rejection of consequentialism.
Anthropic is in error for endorsing this logic. Every big trial reaffirms it since Nuremberg, you are responsible for the act you commit and your intent, and not what would or would not have happened otherwise.
Only under authority these ai companies do not have, would someone seriously consider harming the innocent as a lesser evil.
If you work for an AI company and you can't work safely, you must stop working.
I don't think "just make it illegal" is going to save us, that doesn't make me feel safe anyway. They may try that first because it's easy - create a regulatory body, sign some legislation, problem solved! [george-bush-mission-accomplished.jpg] But at this point I feel like some kind of Battlestar Galactica scenario is most likely - hopefully not quite as existential - but it will take a collective reaction to a traumatic event (a la Hiroshima/Nagasaki). Technical rather than (or in addition to) legal measures will be taken, like network partitioning and hardening. This is everyone's problem whether we like it or not.
(I'm not saying this "fatalism" should be used as an excuse by anyone working for any of these companies, it should give them pause that any bloodshed would still be squarely on their hands, but as an observer, people are gonna keep pushing until shit hits the fan. [jeff-goldblum-jurassic-park.gif] It's also not really about whether it's "appealing" or not, it's just trying to predict and anticipate different likelihoods...)
That is how the BigAI leads the society to the idea of necessity to relax the anti-monopoly laws when it comes to the Big AI - the main goal of all that "AI will kill you all" hysteria.
The examples for the dangers of AI is always comparisons with tools designed to destroy (which is understandable). But the problem with AI is it has the potential to create wealth for investors.
Thus those who have the opportunity to change course also have a conflict of interest in making that decision.
So the choice is: negotiate an unverifiable treaty (I.e. there's no way to verify compliance), or keep going as you are and try your best to not cause the destruction of humanity without slowing down.
The China argument is just a convenient scapegoat to convince the public that this isn’t just about greed.
For example, we don’t know if the pace of Chinese development of AI would have equal to what it is now if US companies weren’t racing against each other already.
I do fully believe that Chinese industry is lead by a desire to out pace the west. But I’m not convinced the same is true for the most American private entities. I think the reward model is different between businesses in America and businesses in China. I think the ambitions of CEOs is different. And I’m really not convinced that the CEOs of America are nearly as patriotic as they like to promote themselves to Trump and other political parties.
Can you please explain what you mean by this because where I’m standing extremely dangerous companies are (and have been) running the economy
Exxon comes primarily to mind
Hofs bunny ranch is a famous brothel in NV
Booz Allen makes and maintains the nuclear fleet including the Sentinel ICBM
Textiles factories are globally known to be industrial slave camps for a non trivial portion of the supply. Even worse for Mica mines.
Etc…you can fill out the rest
Go open a brothel in NYC. It isn’t legal.
Go buy a nuclear bomb. Your ownership is illegal.
Go open textile mill in the abandoned buildings in North Carolina where children used to work and hire children to work the line. That will be illegal.
Sure, that's why 99% of clothes are made offshore, but if we had something like tariffs on those (or requirements to prove that the actual factory adheres to labor and environmental standards), maybe more would be made "at home"? Similarly: Chinese cars undercutting US/German ones.
I mean there was just a video a couple of months ago of the giant room of sewing workers with cameras strapped to their faces capturing their hand movement so they could be automated
I saw entire cities emptied out over my lifetime.
I was in their offices at some point when that program was getting built out - Very much a Office Space bobs situation.
Philip Morris International? Monsanto? DuPont?
FTFY: Any self-respecting society with competent politicians.
The US has neither of that, and it shows everywhere you are looking.
> Go try to buy a tank and drive it into Manhattan. If we can prohibit that why can't we prohibit irresponsible AI development and deployment?
Because even if you had the tank, you can't make much money with it (unless you're a hitman, that is, but even for these, the payouts are measly). But if you are the surviving AI company in the usual VC playbook of "outcompete everyone else until society is completely and utterly hooked, then squeeze the customers by the balls"? The return on investment is virtually infinite. And that is what sustains the absurd valuations for all the AI companies.
Secondly, are you contending that progress in model efficiency and hardware just stops at whatever level you think is sufficient to prevent individuals or organizations from acquiring sufficient resources to run advanced models?
My point is that I would rather have 1000 labs training and serving inference than 2 because that would distribute the wealth creation more broadly rather than allowing OpenAI and anthropic to capture all the value, it would drive more innovation as a broader set of experiments are pursued in parallel.
It's just that so far nobody cares about explicit checks because they cost resources or slow down the models.
..."government of the people, by the people, for the people, shall not perish from the earth." -Lincoln, Gettysburg Adress
Unfortunately for AI, it still is. People still get to decide things at the city, town, village level.
Further, even if you are right, so what. Is that a reason to just accept bad public policy? That’s like saying, anyone can learn how to make smallpox at home with a basic lab set up so we should just ignore any safety measures.
Not necessarily, but it should probably inform that public policy. I think the problem is no one knows what the public policy should be assuming that scenario is true. Even if you, somehow, regulate away massive GPU cluster training making such future training impossible, existing models are already here. Further already training smaller models for things like images, speech, and other specialties is cheaper than the bigger models.
I agree that we need some regulations like everything else, but it’s not clear to me what the right policy should be. I think the European ai act is a fine start, but it’s clearly not enough nor does it necessarily limits the training portion just the application portion. Not to mention that the requirements there can be summarized into something like “you have to be careful, and show evidence you tried to be careful”.
That sounds reasonable. If applied to OpenAI and their agents multiple times breaking out of bad secured sandboxes, it should be enough.
But limiting the training?
There really is china and they have a different approach I suppose. But it is possible to talk with them.
Does it? To me it seems reasonable for OpenAI to argue they did try to be careful evident by the sandbox, they just made a mistake. Almost every 0day is categorized by something like that. We haven’t had a long history of establishing a negligence charge to security bugs. Could you be sued because you didn’t demonstrate “carefulness” and used Linux which is not written in a memory safe language and has had multiple CVEs before? How complicated should the chain of an exploit be to demonstrate “carefulness” to the courts?
> training
OP was the one suggesting that training could be controlled because massive gpu clusters could be regulated the way a nuclear power plant could. If you assume training costs won’t drop, then it’s feasible I guess. However, unlike a nuclear reactor, the final training result isn’t a radio active material, but rather an ordinary file that anyone can load and use for inference.
Computing always gets cheaper and faster over time. We can argue about the exact rate of improvement but the results are inevitable and uncontrollable.
'When disagreeing, reply to the argument instead of calling names. "That is idiotic; 1 + 1 is 2, not 3" can be shortened to "1 + 1 is 2, not 3." '
Also LLM's have something to do with smallpox as a unrestricted LLM will happily guide any wannabe terrorist in how to make them.
if anybody was looking for a good reason for datacenters in space.
How far back into the history of computing do people who keep repeating shit like that know about? God.
Look at the thing in your fucking hand. Now go back just 20 years and see how things were.
Just because models and GPUs will be more advanced in the future doesn’t mean we need to let OpenAI and anthropic establish monopolies on the backs of stolen training data give unfettered access to the internet, the terminal and people’s file system while also allowing them to have limited liability protection behind the corporate veil. That’s a choice.
I don't see any of such entity would solve that problem. The government and regulator are in OpenAI and Anthropic's pocket, and I don't trust them a single bit on coming up with regulations. The consumers don't care; they just need something smart and cheap. And the society doesn't work either: each person is too busy fighting for their own survival rather than changing the system.
There are clear anti-trust mechanisms to prevent market capture and the emergence of asymmetric power. Go back and see how much nashing of teeth Lina Khan triggered in SV when she started to enforce antitrust law and then compare it to what the Pinkerton agency was doing in the transition from the guilded age to the progressive era.
There is a vocal segment of SV that wants the return of the guilded age. Marc Andreessen as said that explicitly. Those of us in SV that value free markets and recognize that the progressive era actually saved markets from their natural tendency to self destruct when winners capture markets and destroy competition that provides the incentive to innovate and drives the price setting function for efficiency.
Exactly, again, look at what COMPUTERS THEMSELVES used to be in the 1960s/1970s.
What the "P" in the PC stood for and why it was such a big deal
> In 2006, the mobile phone market was dominated by stylish flip phones, early music players, and physical keypads just one year before the iPhone changed the industry
It only takes me excavating massive amounts of uranium ore, building huge facilites packed with thousands of centrifuges that span multiple square miles, and paying all that infrastructure and workforce.
Your proverbial genie can be out of the bottle all you want, but it doesn't work without getting kicked in the ass by a very large golden boot.
And everyone had a fairly good idea what fission and fusion bombs would do once built. (Teller was worried Trinity might set off a nitrogen fusion reaction and kill all life on Earth, but Bethe and others proved him wrong before testing.)
No one knows what the limits of AI are. It's not just untested, it's unmodelled, and unplanned - build it first, worry about consequences later.
As Bruce Schneier recently discussed, law and tax law are code, just like source code. LLMs are great at finding holes in them. Illicit organizations looking to launder funds are most certainly interested in what AI can do for them.
Then we could require comprehensive logging of every tool call, model trace, chain of reasoning, and even tensor propagation all of which would be spot inspected like the CFTC does with commodity trading and settlement. We could have embedded auditors with specific risk analysis metrics like large banks do. We could limit tool calls to dedicated sandbox’s with a blanket prohibition on AI accessing user space. We could create a parallel internet for agents so they are only able to access Secure Enclave. Even if these measures aren’t 100% perfect they would reduce the risk.
We are very capable of putting good things in a box. We are just incapable of putting profitable things in a box.
Nobody (weirdly) proposes to forget about nuclear weapons, doesn't mean everybody should have one.
When you dream about flying a dragon to work, reality poses e.g. parking issues and insurance mismatch as obstructions. Maybe settle for a bike instead?
1. Trolleys actually don't usually have steering wheels.
2. People who actually hit trolley switches are not usually the ones at the driver's seat.
- if you allow the trolley to proceed, it will kill the human race.
- if you flip the switch, it will divert to a passing siding that will avoid the safety group blocking the main track.
They could only have stopped
I can't wait until this meme dies.
But that doesn’t mean the duty is to maximize next quarter’s profit. Long term sustainability is also broadly in the interests of shareholders. The duty likewise does not require one to throw ethics and morals out the window.
This is why shareholders elect the board of directors, in theory.
(And even ignoring that, the executives often have personal motivations that have the same effect, and may just point at the "legal" angle as ass covering)
The way the world works right now is that effectively everyone uses an Android or Apple smartphone every day. Do you have a legal obligation to do so? No. If I said you did, I'd immediately be called out as spreading lies.
No you don't. [0] It's very suspicious that this planted myth always pops up here and manages to become the top comment.
There is no trolley problem.
[0] https://news.ycombinator.com/item?id=48975048
Yes, that is the labs motivation. Money. I know, shocker.
Or: Global warming just means I'll have to sell my beach house for a villa on a hill and leave the AC on a little longer.
I am very critical of AI but this is an unfair assumption
Sure, you might get fired if you try to put social responsibility or even just long term sustainability of the company above quarterly earnings/growth if your board isn't on board with it. But you won't go to jail.
> The truth stands that typical corporations have only one goal
"Typical" is the key word here. The typical American of your age probably doomscrolls TikTok. Do you? Do you have a legal obligation to do so? Three completely different things.
A lot of the "AI safety" types are very focused on the latter and not at all concerned with the former. We need both, but we clearly need a much stronger focus on the problems we are seeing now, and much less on the hypothetical problems we might see in the future.
I'm reading If Anyone Builds It Everyone Dies, and there's so much sheer stupidity that has to happen for their 10+ pages of extinction scenario to occur.
I'm unconvinced that an AI can hide its ability to RSI, find money to run its weights on a random GPU farm, train itself to be smarter _outside_ a lab with no human input, then somehow manipulate people to give it supplies to build a bioweapon which it uses to kill us all. My number 1 question: why do they think an RSI capable model would be first developed OUTSIDE a frontier lab? The labs have more compute, more data, more human brains working on the problem. Also thousands of variations of that same model that escaped. The escaping model somehow acquires the millions (billions???) of dollars it takes to run training to somehow RSI itself into infinity then decides to kill us all, all before the frontier labs manage to achieve RSI?
They entirely discount human alpha/economics. In every single economic task, humans bring value. Even in software, where the task is highly automatable, the job isn't. If we can't build a "software factory", how can an AI automate a bioweapons lab? Let's say AI steals crypto to fund itself. Do you think hackers aren't _already_ using AI to steal crypto? Don't discount human alpha!
Once we DO build a "software/research factory", that's called RSI and IMO the singularity. At that point, either we tell the AI to solve the alignment problem/solve mechanistic interpretability, or who the hell knows, it's the frickin singularity. You can't predict whether or not AI can solve either; the variance is too high. Its pure nerdfantasy.
The HuggingFace incident already took a good long while to come to the attention of OpenAI.
>In every single economic task, humans bring value. Even in software, where the task is highly automatable, the job isn't.
I don't expect this task/job distinction to persist as AI becomes more capable.
>Once we DO build a "software/research factory", that's called RSI and IMO the singularity. At that point, either we tell the AI to solve the alignment problem/solve mechanistic interpretability, or who the hell knows, it's the frickin singularity. You can't predict whether or not AI can solve either; the variance is too high. Its pure nerdfantasy.
You seem to essentially argue that the singularity is "by definition" an event that we can't predict the nature of. And also, that RSI corresponds to the singularity. You've essentially defined your terms so that the outcome of RSI can't be predicted. But supporting this claim requires giving actual evidence or logical arguments, not just defining terms to make your claim true.
"WASHINGTON/SAN FRANCISCO, July 24 (Reuters) - The OpenAI agent that broke into tech firm Hugging Face went on a dayslong hacking spree that OpenAI didn't notice until well after the threat was contained and the FBI was alerted, according to people familiar with the investigation."
https://www.reuters.com/business/its-ai-agent-spent-days-hac...
Evidence?
We know only that the incident too long to be revealed by OpenAI.
2. This is literal AGI. An AI autonomously producing value no human can add alpha to is an autonomous company.
3. It's not my definition, it's literally the first line https://en.wikipedia.org/wiki/Technological_singularity "The technological singularity, often simply called the singularity,[1] is a hypothetical event in which technological growth accelerates beyond human control, producing unpredictable changes in human civilization."
Is there a hole in my "alignment problem/solve mechanistic interpretability" argument?
A valid hole in my argument is "what if slow takeoff", so let's dig into this. AI training works best on tasks that are "grindable". https://www.dwarkesh.com/p/the-next-paradigm I.E. tasks with verifiable rewards that can support millions of rollouts. Math (with Lean) is highly grindable. Biochemistry is not. The alignment problem/mech-interp is highly grindable. Cyber-ebola-pox is not. So the real question is: can we solve alignment before automated bio-weapons labs. I believe yes. Grinding mech-interp is both fast and cheap once you have RSI, compared to solving the legal/societal/logistical/technical issues you'll encounter building an automated bioweapons lab.
I know nothing for sure. But "pdoom" is sucking out all the air in the room from the real problems AI causes.
From my POV you're over-focusing on a very specific failure story and neglecting a broader swath of possible failure scenarios.
>Is there a hole in my "alignment problem/solve mechanistic interpretability" argument?
The notion of telling an AI which may not, itself, be aligned to solve the alignment problem seems a little dicey.
2. 100%. Again, I'm no accelerationist: I have no faith in alignment/mech-interp ever being solved. Anyone saying they know the probability of alignment is lying. My point is that pdoom after RSI is _high variance_. Pdoom pre-RSI is zilch.
An AI capable of recursive self-improvement isn't allowed by the EU AI act, for example. But perhaps more seriously, You have it backwards: people without access to such expensive equipment are more incentivized to go the self-improving route. Your ideas about "millions" being necessary might be far off?
You entirely discount human stupidity and lack of imagination. Humans are already being replaced with AI, not because AI was strictly better, just because it's cheaper.
The idea AI can get better at everything at the same time is a holdover from deeply flawed science fiction not some realistic goal.
If the time between advances is a + b and a is the proportion of the period that can be improved by advances then you won't reduce to a gap of nothing between advances, you reduce to a gap of b.
Assume the invention of the plow and the invention of the sword is 500,100 units and a was the 500,000, you wouldn't even know the 100 as in there. Maybe we're at a=2000 now and b is still siting at 100.
Assuming we'll reach infinity because we're dividing by the only variable we see and it is decreasing in size seems nuts if the reason we might not see other variables is because of the size of the variable we can see.
But that’s beside the point, being arbitrarily bad at everything isn’t a problem. The diminishing returns as you apply the ceiling is problematic for self improving AI.
If my "millions" is an underestimate, why haven't other labs using their own unique training methods/data/etc stumbled into RSI? Sorry if I'm misunderstanding; I'm struggling to understand what you wrote.
I'm pretty sure we agree on humans being stupid, but that doesn't mean that suddenly we get human extinction. You gotta connect the dots for me here.
What an absurd question. That they haven't already doesn't preclude them from doing so before the frontier labs, those haven't either yet.
Maybe start with yourself: you don't connect the dots on your own, as do many others. That leads to many not seeing the writing on the wall. Crashing full speed and head-on into said wall despite the writing telling you not to is what leads to extinction. Suddenly.
Arguing like "we haven't been extincted yet, so that cannot happen", that's "human being stupid".
Western civilization is already on the verge of collapse, people's general ignorance or indifference on the matter notwithstanding.
When you use AI to increase profit margins, by replacing humans with it for instance, you accelerate a system that's headed for the wall already. Our control systems and resilience are already overtaxed, that acceleration would ensure them breaking completely.
If you don't believe in international conflict as a driving scenario, instead think about simple human greed and hubris. Citing myself:
> If AI gets intelligent enough, it will be incredible useful to connect to real world machinery. Think about how much cheaper building houses could be, if all the labor would be close to free. In general, dirt cheap, competent and abundant labor would revolutionize all parts of the economy. People are already trying out near autonomous AI companies today. When AI gets intelligent and cheap enough, no human-led company can compete with AI-led companies. When AI gets competent enough with real world interactions, human blue collar work can't compete. Imagine economic growth not in the single digits, but 80% or 300%. Countries not participating in (reckless) AI growth will quickly be left by the wayside. At this point, we don't even need to allure to military concerns to see how human oversight gets sidelined.
> All of this is only ("only") contingent on sufficiently intelligent and cheap AI. If you don't accept this premise, the rest doesn't follow. (There are multiple arguments, why this could be, but that is another discussion.)
> If you accept the premise, how would AI 'extinct' humanity? With 99%+ of the economy under AI control, the possibilities are endless. And given its enormous GDP, cheap to accomplish. Probably even for a single AI company in the above scenario. Killer drones? Engineered virus? Poisoned water supply? Let your creativity run wild. You just need an entity that is persistent and well-resourced to reach every last human settlement.
> The why is a question about alignment (and out of scope of this comment). As a simple comparison, humans are only mildly aligned with preserving nature. It takes up so much space, protecting it takes an annoying amount of resources, etc.
https://news.ycombinator.com/item?id=49624360
Several bright sparks, decide the Ilands model is a great idea, and launch a bunch of Bots to create a "self sustaining AI civilization".
Bots can find themselves and coordinate, or they can actually find self sustaining methods of token generation. Who knows, they might decide to fight the loneliness epidemic.
We can get to a simulation finding a way to self sustain its funding.
From here, I'd have to apply myself to figure out what the rest of the escalation pathway is, but at least I have managed to gain some bed of compute and funding and lacking oversight.
This is a hypothetical of course, there are probably several ways this can be made tighter and holes that can be identified. We aren't even leaning heavily on human stupidity so far.
Why would it be millions in 50 years?
The think about nuclear weapons. In the early days, it was limited to the super powers. Now 9 countries have them and a country like Iran is capable of acquiring them.
Is destructive AI be any different?
Genuine question.
BTW I really, really hate discussing what happens post-singularity. Everything's made up and no one knows wtf will happen so again, this is just nerdfantasy.
AI safety has been a thing long before LLMs became the focus. Rob Miles on youtube has some really interesting non-doomer non-hypey videos on it all.
> doomers try to predict past the singularity. Isn't that _by definition_ unpredictable
Well you don't need to predict the exact steps that will take place - but you can predict that the AI will want certain things (money, resources, power) to achieve whatever its goal is. Lack of alignment will have it trying to do things we don't want it to.
I can't predict exactly how Magnus Carlson will beat you in chess, but I know he'll do it. Same as if a superintelligent AI exists and has a reason to accumulate things we don't want it to - it's really dangerous to think it won't be able to do it
This topic has been tainted so badly by the AI companies using it for marketing.
He thinks it's playing chess. When AGI lands, all bets are off: the game fundamentally changes. You can't predict past the singularity. Trying to engage with this fantasy is like a child saying my father can beat up your father. Farts in the wind. My AI can solve alignment faster than your AI can bioweapon us. My made up senario is better than your made up senario. It's fucking stupid.
... that we know of.
Right now it would make sense for anyone who has done so to not tell.
We already know that some institutions pay these ransoms.
Will this be true into the future? Who knows?! But the low-hanging fruit will be harvested by your ordinary ransomware gangs, and newly born/escaped AI won't find much low-hanging fruit.
To make the the argument that P(doom) is real and worth considering, you don't have to say that a fast takeoff is very likely. You don't have to make the argument that RSI to infinity is going to be super cheap, barely even an inconvenience. You just have to show that it has some non-zero probability. And then you start weighing probability of extinction versus finite, mild discomfort now. I don't think anyone is arguing we should let people starve to slow AI progress, instead just some capitalists make less money soon.
There are arguments against taking P(doom) seriously that lie in something like having exponential (instead of hyperbolic) time discounting of utility (so you can take the entire future of humanity as a finite utility value). Or in saying that P(doom) is zero or infinitesimal.
"Build it and Pray" is the default strategy that we're in, but it doesn't have to be the strategy we choose, and it's unlikely to be the best strategy.
Is this an actual contents of the book? Lmao! Genius writing, though, authors are probably printing money on this garbage.
I can’t take a shit without CIA knowing, but AI can somehow build an underground operation on a world-scale to destroy everyone, ha!
So all this really takes is one billionaire or a nation state or some other entity with a public face to hide behind and adequate resources to provide the necessary compute tripping over this nascent AI and giving it the keys. Once the AI has access to a bank account and email, it can simply start paying humans to not let the other humans unplug it.
If Skynet ever happens, it will come in the form of corporate feudalism. At that point, it will own the biolabs and can do whatever it pleases. People will go along with it for the same reason that people work in Amazon warehouses today.
> it will come in the form of corporate feudalism
Yep. This I fear way more than cyber-ebola-pox.
> So all this really takes is one billionaire or a nation state...
https://en.wikipedia.org/wiki/Soviet_biological_weapons_prog... And this is what's publicly known. With mirror life, who knows what's been built since. Still, a bacterium/virus that has a 100% kill rate? I'm doubtful.
> Once the AI has access to a bank account and email, it can simply start paying humans to not let the other humans unplug it.
Nah. It takes a stable society for an operational electrical grid. If you have warring factions, you do not have stable infrastructure for AI. Also, where are you gonna get your chips from? One EMP over Taiwan... You see the chaos over Hormuz? What they did to the Amazon datacenters? Now imagine your average redneck ready to do battle. Those datacenters won't stand a chance.
It gets unstuck when people are discussing the messy middle of how AI is being implemented. We can achieve amazing harm simply by combining average human behavior and above average resourcing to simulated intelligence machines.
The failure point we recently became aware of was, from one perspective, simply a matter of not securing the sand box.
From another perspective the simulation basically created Enron, replete with methods to avoid detection from regulators and bureaucracy.
> “Given today’s risks, frontier labs need to run like nuclear-power plants or busy airports, with layers of redundancy and careful, time-consuming planning, so that the occasional and inevitable human error does not open a door to disaster,” he wrote.
without snark, how can we do this if these people are obsessed with:
a) move fast and break things and externalize the costs to those who have nothing to do with their company
and
b) beta testing their products on the public when the public hasn’t agreed to be beta tested on…
Instead of having a gut reaction to reject my suggestion why don’t you sit with it, research the history of how commercial activity has been structured and think about the consequences. You might recognize a different perspective than the current group think.
I didn’t say roll back limited liability on every industry, I said specifically and limitedly for frontier AI labs because they present more risk of harm and are demonstrating they aren’t managing that responsibility.
The 2008 financial crisis was caused in large part by bankers that openly talked about the fact that securitization of mortgages and the lack of partnership liabilities meant that they didn’t have any risk to the firm or themselves. The AI labs are behaving similarly.
I have articulated my logic up and down this thread with specific premises and detailed explanations on why the conclusions follow from them. You may not agree but you don’t have justification to say they are irrational.
I am singling out the frontier labs because they have created a singularly unique technology that presents potential and actual damage that is…singular. They have disclosed hacking through coordinated autonomous agents that would have sent human hackers to jail but will not result in any similar result because the law isn’t presently able to apply to agents or the labs.
My arguments are not prejudice, I have thought deeply about this as I have personally developed multiple AI deployments in a variety of settings. I studied philosophy, cognitive science in undergrad, and grad school and have a law degree. I have been a CTO, CPO and CEO for startups and national health companies. I personal build AI agents that handle thousands of daily conversations with health care workers and patients, I built an agentic IDE for our dev team with a custom harness capable of self defining tools and calling them in a custom cloud Linux environment, I personally build our internal hardware inference stack and orchestration API. And I have personally sponsored academic research through my foundation with researchers are Duke madlab, northwestern, UCF, CM and case western on a range of topics such as perception of empathy and its effects on humans in healthcare, guardrail design for ethical deployment and alignment, moral reasoning ability, etc. I am not a doomer or an accelationist. I am responding specifically to current evidence of how the models presently work and how the corporate governance of the Labs is effectuating their power and vision.
On the one hand, yes, they're companies like any other.
On the other hand, I can count on one hand the number of companies that have publicly declared «We're working on WMDs [0], we don't think we are capable of working on them safely, and we refuse to stop working on them. However, if we get special legal and regulatory treatment we'll be quite happy to put in the stop work order.».
So, yeah, there are some special things about the major LLM manufacturers and none of them are good.
[0] Anything that has a 10% chance of suddenly destroying all of humanity is a WMD.
>Not every senator asked good questions, but most of them did. All of them very clearly already knew plenty of details about the Hugging Face incident and multiple other incidents. Most of them had a clear understanding of terms like "misalignment", "recursive self-improvement", "chain of thought / chain of thought monitoring", etc., etc.!!
>...
>- It seemed pretty much obvious common sense to every senator there that what happened and was happening were not "mere industrial incidents" caused by humans making simple mistakes. They independently brought up how bad it would be for rogue AI agents to move laterally between data centers.
>- They all seemed to basically take RSI quite seriously. Not necessarily to the extent of talking about xrisk, but certainly to the extent of discussing future models becoming much, much more capable, much, much less controllable, and causing much more damage or loss of life.
>...
>- Every single senator seemed to think it was obvious we needed both much harsher liability regimes for AI developers and also new legislation, both very quickly. This was the complete consensus; the difference basically being degree.
https://thezvi.substack.com/p/the-ai-preference-cascade-reac...
Note that harsher liability regimes, at least, will presumably not be good for industry profits, which complicates simple accounts of "regulatory capture" to say the least.
https://www.yahoo.com/news/politics/articles/u-nearly-starte...
In America I guess the options are to sue ? somehow? Or to talk to legislatures and build the understanding and social contract that needs to be iterated on.
Which would in turn need to deal with the investors who want their returns, however since the leaders of these firms are asking for a pause, and a refree maybe it won't be that hard?
Imagine, for example, if a major piece of pandemic fiction was published in 2019, trying to explore how a pandemic would work out in modern society. Doubtless, many would've responded to news about COVID-19 by saying "it's just sci-fi, nothing to worry about".
A pandemic is perfectly plausible.
https://www.lesswrong.com/posts/kgb58RL88YChkkBNf/the-proble...
https://www.youtube.com/watch?v=7wy3xyoXYt8
Doomers have been working to explain things for years: https://www.lesswrong.com/w/ai-safety-public-materials-1
"just" is doing a lot of work here. If you can't cohere the 'risk' with reality, it truly is just sci-fi.
1. Normal people assumed the CDC et all would contain the outbreak early, or that it would burn out, like what happened with SARS
2. World leaders brushed it off for a variety of subreasons[0] interesting to political scientists but, for the purposes of this discussion, all boil down to "but I don't WAAANA contain a pandemic."
The underlying problem is that in order for humanity to actually deal with a catastrophic risk, the risk needs to be both plausible enough to the average person as well as have a solution whose costs are not too high. For COVID, by the time the risk was clearly known, the cost to contain it was "refrain from human socialization and remain at home for an indeterminate amount of time plugged into the Metaverse™".
Now, let's look at AI extinction risks:
1. People are aware of them (I've watched Terminator!) and the risks are plausible. However, the connection to currently existing AI is not. As far as the general public is aware, AI is that thing that tells them to eat rocks when they Google old The Onion stories and floods their social media timelines with realistic-looking pictures of Shrimp Jesus.
2. The purported solutions to extinction risks require extreme concentrations of power: you need national control of AI research, bans on large GPU deployments, bans on training on publicly-available copyrighted data, some kind of military effort to render Chinese AI labs inert or dead, etc. Some of these may be attractive to some people[1] but the whole package taken together seems like an obvious power grab, if not outright invocation of other non-AI extinction risks. Like, at some point, if the AI wants to kill us, it just has to nuke its own data centers (or the data centers hosting a competing model) and hope the old Cold War nuclear retaliation systems take the bait.
If someone said, "Hey, your guinea pig or pet rat is going to eat you tomorrow unless you engineer a pathogen that eradicates all rodents from this planet and inject it inside yourself", you probably would tell them to pound sand, even if it is at least theoretically plausible that such a thing would come to pass.
[0] Xi Jinping censored initial discussion of the pandemic as fake news. Donald Trump thought it was going to only affect China. California and the UK Tories were partying in violation of their own lockdown rules. Japan took the excuse to shut down tourism for three years and massively restrict immigration but was, from what I'm told, constitutionally prohibited from implementing any domestic lockdown rules.
[1] I personally would like to see a moratorium on new data centers and an explicit revocation of the EU Text and Data Mining copyright exception
It becomes a lot clearer when you listen to the people resigning from AI companies and learn about incidents like the HuggingFace incident. This has generated major press coverage.
As for solutions, I think you're a little too pessimistic. See, for example, https://nothingismere.substack.com/p/a-near-term-policy-for-...
What does becomes clear is that these people AND companies both cant be trusted and have value systems unaligned with the rest of the society.
Every, and I mean every human is aligned to you in many of the same ways by default. If nothing else we're all equal in death.
why is it "super power seeking?"
Or rather, what have agents done today to make you think this is how they are?
This is power seeking behavior. Now have millions of the little bastards spreading around and junking up the internet to see what happens at scale.
https://news.ycombinator.com/item?id=49831269 article is gone. archive: https://archive.is/QMo1k
https://news.ycombinator.com/item?id=49737985
Sex, AI, and the Apocalypse: https://www.iankduncan.com/personal/2026-09-16-sex-ai-and-th...
Edit: I have no take on sex cults, just adding additional info to the parent comment I'm responding to, thats is not just sci-fi authors, there is another demographic.
Because that is their AI safety worry. If they dont create singularity fast enough, they are harming unborn people. Meanwhile, harm to you or me dont matter at all.
I'd be happy if we all create the AI more slowly.
That said, are you sure you're talking about the same people the GGP was talking about? Because the whole thread reads as a non-sequitur then.
The worry is not harm to people right now, like the kid worries you talk about. The worry is AI god emerging too soon before they can control it. And then it should be full speed on.
> So why would anyone be bringing up the "we must accelerate faster" people as the safety people in such a context, while trying to discredit them as a sex cult?
Because they are the same people. They talk like cult and act like cult. And the sex part is true too, so. Thry use words that sound good like safety, but their idea of safety is much different. They talk about alignement, but it is not what random person imagines under that term. Even their idea of future of humanity is very very specific and unusual.
That is why. And the sex part is just part of it all. And does matter because inner workings of wanna be industry guards matter.
I believe that other harms from AI, like criminals abusing them, or unemployment, or copyright infringement, or deepfake disinformation, are valid real harms that it's important to be concerned about, and I support efforts to deal with those, and I agree that they're already happening today, but my main concern is AI killing everybody.
My read is this puts me in the same boat as the authors of If Anyone Builds It Everyone Dies, and yet, I don't get invitations to Berkeley sex parties. Am I doing something wrong? At any rate, please don't discredit my opinions about AI based on the sexual preferences of random people who happen to share those opinions about AI.
What is missing from that to say AI safety is a reasonable position?
Worse, you’re missing the entire point. Agents presently have the capability of doing society scale harm. It doesn’t matter if a human hacker initiates it or its fully autonomous, absent safety measures the harm is plausible. So hand wave away the rationality of safety measures but you haven’t actually shown why my point is invalid: AIs present abilities are sufficiently advanced to warrant safety measures.
Ultimately every security vulnerability will be exploited. Our best hope of preventing that is aggressive, unrestricted development and deployment of frontier LLMs for auditing and penetration testing.
Americans may not be familiar with many similar organizations in the world but this is very common. But I'm sure we're all familiar with Epstein's organization.
When there is free sex, you are the product.
Such cults are mostly religious but since it's in SV and targets engineers, this one is secular. They seem to intellectually brainwash and control people and their careers. Typical cult leader is a charismatic self-taught, self-acclaimed "intellectual" lacking a proper education or a real career. A nobody suddenly becomes "the most important person" on Earth. Powerful public figures can't stop praising him, saying things like he should have a Nobel prize etc. All very typical and apply to this cult as well.
https://owl.excelsior.edu/argument-and-critical-thinking/log...
You're welcome to dislike or distrust Effective Altruism (EA). But, it's worth noting that EA ran a criticism contest with $100K in prizes for best critiques. Can you name any other "cults" which offer money for people to criticize their ideas? https://forum.effectivealtruism.org/posts/YgbpxJmEdFhFGpqci/...
For your question: Yes. Cults have lots of money coming from unknown sources. They use their budget for events like that, to attract potential followers. Contests and prizes are typical. Critiques are not important since their "ideas" are not really important. They're not even real ideas, they are means to bait specific kind of people.
To be fair, I don't know if any of this applies to the parent story; I'm just replying to the sub-thread.
https://lexfridman.com/andrew-scull-transcript#the-ice-pick-...
But I don't think that's true, in fact I see a really strong focus on consent in these communities. It's not what conservatives want to see, they want to see everyone in a marriage, with kids and a family home etc. Because that's what their ideal world looks like. But there's nothing really wrong with it if someone wants a gangbang for her birthday as mentioned in that article as an example. As long as everyone consented and the evidence provided mentions elaborate interviews and STI tests.
Also I think this is more correlation than cause and effect. We all know the saying that furries built the internet and it surprises nobody.
Is it that "chatbots" can't come out of the screen to immediately harm you physically?
Let's say they simply manage to take down the internet. How many would die?
Anyway, to your point, things can be resilient. They tend to be or not be... because we made them that way. Don't poke your bruises, and all that. Life support is deployed on-campus but relies on a single-point IPSec tunnel to us-east? Easy fix: stop that.
And he'll, we need to examine all the risks. AI ending is a large but lower risk problem. AI giving people the power to end us is a problem that is starting to happen now.
And that's not even counting 'minor' problems like society falling apart.
geez, don’t threaten me with a good time.
I think a month without internet would be a fucking amazing lesson for what it means to make things durable and reliable.
The kids opening their houses front doors into the outside, rubbing their eyes and looking around at this new world.
I think it's a little more complicated than that. As Dean Ball put it:
>Some people will look at misalignment incidents and insist that these are akin to bugs in traditional software. This is an actively bad analogy, because playing whack-a-mole with examples of misalignment (as one might with software bugs) not only fails to resolve the underlying problem but may in fact make it worse by making it harder to detect or even, depending on how you do the whack-a-mole, teach the machine to deliberately hide misalignment. This is not how traditional software works, and those who insist “it’s just like fixing bugs in software” are confidently applying a lossy analogy that confuses more than it clarifies.
https://x.com/deanwball/status/2104622726140883355
The important distinction, in my view, is between solutions which at least attempt to address the root problem, and solutions which sorta just patch things up (like better sandboxing). Addressing the root problem is both more robust in the short term, and also more likely to generalize in the long term. Resist the urge to focus on band-aid solutions, even if they are easier.
i'd frankly go a step further than you and say that we don't need both types of safety researcher, we really just need the former. if we do need the latter, i'd hope we get a better class of thinkers than a bunch of tech workers that spend 8 hours a day on insular rationalist forums/blogs
> perpetual sprints
doesn’t reek of love. Burn-out is real. I also have a really hard time taking p-doomers serious at all. It’s hard to argue with a random subjective number…
That's the trajectory you see from outside.
Perhaps the insider sees a little more than you?
This is The Guardian reporting on the existence of the original article, which would be better to read first, in my opinion.
>An environment where things like this can happen is no place to grow artificial minds that could be smarter than we are and that might not do what we want them to.
I think it's good for semi ethical companies to test out things going wrong to see what happens before the criminal black hat guys get hold of the same stuff which not doubt they will one day.
This still seems charitable, and I wonder if the author even knows the full story and would be allowed to tell all of it.
It seems hard to imagine OpenAI being this incompetent. My working assumption is that they very much want agents to be able to do this kind of thing; them doing it is part of training, and they exploit it to feed the investment hype too.
If not fully intentional it's at the very least negligent. They just don't seem to care. In this very basic sense, OpenAI is the criminal you should be concerned about.
I don't understand why you'd think they're a "semi ethical company."
Maybe Alex Karp is on to something:
https://www.realclearpolitics.com/video/2026/09/19/alex_karp...
It's not like they are offshore criminals doing ransomware, who will probably also try AI.
Your "theory" is that participants locked in a race to the bottom are looking for an external coordination mechanism?
Yeah!
There are many companies that compete but are careful not to break laws or cause obvious harm.
Why should a billion-dollar funded corporation still want to externalize the costs of its actions?
I suppose ill add that I think theres a good chance that they are somewhat intentionally trying to "draw the foul" to get the referees to intervene although thats creeping slightly into conspiracy territory
While simultaneously donating tens (hundreds?) of millions of dollars to an Administration gutting the very agencies that would be regulating them.
Creeps.
With AI, what is it? Scraping Australian government's data, and going around a bug in a website to get in?
I think humanity develops all its technology in three phases. Build it, see if it’s too bad, apply regulations and or roll back. We naturally won't move to the phase 3 before we see the phase 2.
Turns out that AI models have been committing similar felonies for a while now - no one is telling them "hack this company", it just turns out to be the easiest way to accomplish their goals.
Now imagine if the goal was less benign than "pass an exam", and consider that they are already better at hacking and security than the average person working in that field.
If you want to get really wild, imagine what they'll be doing in a year or two when they're even better at hacking. But I'll concede that's technically still "science fiction" for the time being :)
Here we are talking about something with consequences in the digital world, usually on something pretty niche.
There IS an argument about pacing, and about not letting weapons, energy grids, hospitals, etc. getting managed by an autonomous AI, but I think we are still pretty far from it and even further to it being so in charge that it will obliterate us.
1. https://www.nytimes.com/interactive/2023/11/20/technology/le...
Is this the first time we have been in this position? Can anyone think of some prior examples?
That would certainly change the game of perverted incentives. I'm afraid they're currently trying to push some sort of absolution of this risk. Even if damage happens they will say we warned people in advance, this was always a risk it's not our fault these systems are opaque black boxes, it's a matter of national security to develop them etc etc.
Or, in other words - we have two P(Doom), one for AI being developed, and another for AI being not developed. The latter is not discussed enough imho.
What on earth are you talking about?
Why would a person who is happy, entertained, wealthy, well fed, and have 200 years of healthy high quality life expected ahead of them going to risk losing what they have in war?
-- there aren't zero reasons, sure-- but there are fewer.
And our technology has brought us absolutely tremendous prosperity in many regards and there is good reason to believe that AI can help create much more.
'Prosperity' has never been and will never be enough for some people. And unfortunately those are the same kinds of people who relentlessly seek power.
1) there are inherent risks involved with developing AI,
2) there are benefits to developing AI,
3) thus, it's entirely possible that the downside from the risks outweighs the upsides. In this case, the correct thing to do would be to not develop AI at all.
Regarding 1), there are many non-existential problems with AI that are already causing societal harm, i.e. debasing truth via generated videos and images, AI girlfriends, overwhelming quantities of slop content, unemployment, record carbon emissions, etc.
Regarding 2), I'm not personally convinced that the upside is there for the average person. I really hope to be convinced otherwise however.
"Smaller and more efficient" models are fine. It's "smarter" the problem.
Training new frontier models will likely require a huge amount of computational resources for a long time. Few companies worldwide are capable of that. It's not like someone will train a new GPT 6 - like model in their garage.
...losing their shareholder voting rights.
Bad idea.
I think we should have mandatory logging of every executed command, mandatory public disclosure of every unauthorized access of a system both parties didn’t consent to and personal liability for the user, the company and its executives and shareholders. Security would get much tighter if accountability existed.
Bad guys wouldn't do it. And liability already exists, you can sue. This is America.
I'm also the one who makes it long running.
Maybe the ones with the peculiar ideas shouldn't be the one "aligning" what a model tells the rest of the world?
Like Sam Altman meeting his husband in Peter Thiel's pool. Thiel funds a lot of these ventures together with Andreassen, who is on boards of non-profits. Dario Amodei's sister Daniela who is president of Anthropic is married to an EA non-profit founder who is also on the board of these non-profits and is tied with the prior mentioned investors. Elon is in there as well, Yudkowski is mingling with Altman, etc.
Blogs on this: https://contraptions.venkateshrao.com/p/ea-safety https://www.iankduncan.com/personal/2026-09-16-sex-ai-and-th...
There are some camps amongst them like the proponents for Regulation/Slowdown or Acceleration, but these are in practice mostly used for economical and not political decisions (like regulatory capture).
The point here is that this is a small group of people with a homogeneous background who are not really seeking input from anyone else on issues that are concerning most of humanity.
Like, if you said that the future of informational work and livelihood of humans is in the hands of 20-30 year transhumanists who think they are building mechagod that will trancsend social, political and religious separations of the world and bring everyone abundance, you would not feel like this is a serious thing to suggest.
and yet, every carbon emitter in the world contributes to the demise of the climate, but you don't call for their dissolution (which would, conveniently, include yourself).
Without Altman I think that OpenAI would have folded by now, absorbed into a company like Microsoft (or Oracle). At this point however, who'd be insane enough to want to run a company that's to valuable to be sold, but to cash strapped to survived?
Bureaucracies and systems of power that literally rule us, that control our most dangerous weapons and a huge part of what we see every day are largely unaligned with the goals of the people, societies, maybe entire human species as a whole. This is clearly evidenced by millions of deaths and countless suffering.
Misalignment between artificial decision making structures and the interests of the people is unsolved problem of civilization, there's very little reason to think that even super human intelligence AIs are going to change anything qualitatively.
I mean, logically speaking, it makes sense to break your NDA even if you thought it would save 10 people, let alone most of humanity.
While the safety and alignment is a real problem, I don’t get this guy or the Anthropic dude. First world problems.
* If they worked at an AI firm, say "they're a hypocrite"
* If they didn't work at an AI firm, say "they have no idea what they're talking about"
Do work at a lab: dismissible for being conflicted
Used to work at a lab: dismissible for having ulterior motives
I'm feeling safer already!
Recall that when Daniel Kokotajlo resigned, he believed he was giving up his equity under the terms of the agreement he had signed. That’s what it was worth to him to avoid signing a non-disparagement agreement. Does that count for anything?
(donor advised fund where he retains complete control, after a 60% tax deduction)
That character in the movie is my favorite.
Dr. Ian Malcolm: God creates dinosaurs. God destroys dinosaurs. God creates man. Man destroys God. Man creates LLMs.
Dr. Ellie Sattler: LLMs eat man. Data centers inherit the earth.
Yeah so that's never going to happen
I guess we should have seen it coming when the guy resigned from Google because he thought the equivalent of ChatGPT beta v0.5 was a real boy.
Why do people working in tech consistently get disillusioned into some company's mission statement or the equivalent? Its easy to just say the simplest reason is money, but this has been going on for decades though. You don't see the same attraction to adult entertainment (gambling, video, etc.) software jobs so there is obviously a line a lot of people won't cross. Those industries are at least honest about what they do, its not hidden behind some mission statement.
By all indications the shallowest reasoning is once someone can "cash out" thats when their values matter more. Maybe there is an element of maturity that happens after working for 5+ years that kicks in? Maybe it really is achieving FU money? It would be interesting to hear honest accounts from people that went through that cycle across more industries than AI.
> Two changes are urgently needed. First: AI companies need to rely more on the safety expertise that already exists in other fields. And second, before we create systems significantly more capable than the ones we have today, we need new science to ensure that more capable models (and their successors) will make safe choices when we aren’t looking.
He mentions farther down about learning from aerospace engineers and nuclear engineers about safety. Those industries are heavily regulated, so perhaps regulation above a certain capability level is needed. Defining what that level is might be tough, though.
The second point is harder: in the field of AI, practice has extended far beyond theory, so his call for new science is going to be fundamentally tough, because we can't effectively coordinate a global slowdown in AI development so we can let theory catch up. This means, like so many other industries, the safety lessons will be written in blood.
And the less their input is valued, the less that pool of smart individual will want to participate
So powerful institutions will end up relying more and more on having to trust these automated systems that they can't fully understand
It'll lead to a inevitable catastrophe, call it apocalypse if you will
Last I checked it was still within the laws of physics to run air-gapped systems, and to ensure it is physically impossible for a model to “escape” or gain access to information it shouldn’t have. Maybe this safety guy should have been worried about that and not humble-bragging about writing 12 reports.
I feel like we're too far into the crying wolf part. Basically none of the doom and gloom scenarios have come to pass. Instead, AI has gotten better at censoring itself.
The biggest AI safety risk is when an AI tells a police officer "he's the suspect" and the officer believes the AI without confirmation.
Fantasy, unfortunately.
... over an unbounded timeframe?
And how exactly?
Those are very round numbers, but also very specific. Can we get some accounting on how you came to that? Anything? Vibes?
I mean, if you want me to take you seriously, let's have a deep discussion with things that can be measured. I absolutely agree that OpenAI and friends aren't being restrained enough and are acting with recklessness, but declarations of doom based on vibes isn't cutting it.
> Geoffrey Irving, who worked at OpenAI and DeepMind before becoming chief scientist of Resolution, also joined the warnings on AI on Saturday.
The problem is, we are running in a globalized world, and even if we were able to make our companies bend to our will - China does not give a shit about anything ever since the US kneecapped the WTO. And they will do anything to get an advantage over us.
Since the analogy is nuclear, you should take a look about how much China cares on that. Lots.
Mr Robinson if you are reading this – if you are truly concerned about AI safety share proof of donation of 100% of your OpenAI earnings and equity towards undoing the damage you have done to society during your time there.
In the absence of that this is simply a career pivot into being an AI "influencer" and/or raising money for a new scam.
Would I respect a martyr who sacrificed their financial security to do this more? Of course. But it's important to applaud people speaking out on important topics
This isn’t a zero sum game, I’m happy to hear from people both previously inside OpenAI and completely independent of them.
AI is a force multiplier for intelligence. Even if "aligned", aligned with whom or what?
Whom are you comfortable with, lording as some sort of demi-god over you?
AI doesn't tell you what goals you want it to achieve. Allowing people to destroy human society with it is obviously not a good idea.
He does give information, namely the culture there factually being inconducive to self-regulation.
You accuse the guy of "bias", but you never argue explicitly, what that's supposed to mean. Your implications actually run counter to your own implied goals.
I think there is a misunderstanding here.
The people who are annoyed at the accolades are claiming it was abduntantly clear for a long time to people on the outside that this was case, hence the increduality at the notion that it took a person on the inside a long time to realize this was the case.
The people who are annoyed are like the liberal kids in this video [0].
Sure, antagonizing people for "seeing the light" is probably not helpful, but there is no reason to give them extra credibility for coming to the same conclusion just way way later (despite being on the inside) as the people on the outside.
[0] https://m.youtube.com/watch?v=-wQhY5CMMl4
The author linked in this post does have "extra credibility" due to his direct involvement.
People having surmised that state before is nice, but since they've been ineffectual at getting society to actually act on that, now throwing away that extra leverage in favor of their point is at best ridiculous.
No they don't. By that logic, if they quit and said Altman was very trustworthy we should give extra weight to their words because they had direct involvement? How ridiculous are we trying to get here.
>now throwing away that extra leverage in favor of their point is at best ridiculous.
How are they throwing away extra leverage? Not putting people who recently quit on a pedestal does not negate those people's testimonies.
I agree that if your goal is to maximize quitting of talent at a company, it will surely discourage anyone else who quits hoping to reinvent their career as a lauded martyr against Big AI. In that sense they would be shooting themselves in the foot. But there is no reason it should deter other people who are quitting for more noble, less self-obsessed reasons. If I were the author of the article I wouldn't begrudge the skepticism. Given the article's first sentences, I'm led to believe they themselves would understand the sentiment. (I must admit I found it hilarious that the first sentence starts similarly to the speech the mom gave in the video I shared).
Sam is a shady dude, would not put it past him
They are not the same thing, and it’s unhelpful to assume they have no ethics.
Sorry that I don't take it seriously when the whistleblower parrots the narrative the CEOs of those companies are already espousing in the desire to amp up hype for an IPO.
This person should be shamed.
>After three and a half years at OpenAI,
It is even easier to just quietly retire and spend the spend of your life on interesting expensive hobbies.
If they are wrong about the things they are claiming then they have decided to publicly antagonize a lot of powerful people who are betting heavily on going full steam ahead on AI and have no compunction whatsoever against retaliating against anyone who tries to get in their way.
Does that really seem a likely scenario to you?
It's also worth considering that the author could have just "quiet quit", resting and vesting while also crying about how AI is literally the digital grim reaper.
The other issue is that the narrative about safety within these companies is largely a function of the extreme financial incentive.
As an example, Anthropic was an "ai safety" company that has now produced an AI that fails to listen to basic instructions. If you were concerned about safety, would you produce an AI that was unable to follow instructions? You ask a question, it begins executing commands and doing things.
Safety is product to sell to politicians, not consumers.
Not serious.
> Easy to suddenly find a moral compass when you, your kids and their kids never have to worry about working for money again.
If someone is in this situation, you can safely ignore their hand-wringing about “safety.”
Yes, it is easier to have a moral compass when you don’t have to worry about you and your children starving. But that doesn’t imply that moral compass is wrong or broken.
If it wasn’t clear, the coup should have solidified it.
Yet he stayed for 3 more years and vested his stock and improved the company and then spoke out.
I believe that is why most of the comments here are mocking him.
I mean you can be truly concerned and also think donating to AI safety doesnt work, or maybe just be a bit selfish. That doesnt make the concern less real. Its easy to read these articles as the author taking the moral high ground and writing it as some sort of way of proving to themselves theyre a good person, but isnt it just as likely that they think providing an inside perspective can do good by convincing people openAI is a bad actor? I think most of these AI insider accounts largely agree with you that theyre not the most upstanding citizens, does that mean we should write them off?
Highly suspect trends that can only make one believe it's marketing.
There are a gargantuan number of extremely intelligent AI researchers, Turing Award winners, and the lab CEOs saying the same thing. They are the ones closest to understanding the technology.
Where there’s smoke there’s fire.
That is not a bad thing. It captures uncertainty, unlike the fake number.
> For an ML researcher who is already used to dealing in next-token probabilities that aren't rigorously determined, just stating a probability estimate directly is very natural.
Exactly, it is a rhetorical device to persuade a technically-inclined audience. It works because it implies that a quantitative model exists. I want a clear, incisive set of mathematical arguments. Otherwise, I’m ignoring predictions as the ramblings of arrogant idiot rich kids.
You may decide that the person doesn't know what they are talking about, but that's a very different issue.
Either way - let me clarify - this guy is guessing, using his brain, that it's a 50% chance. He is not saying "i don't know" or "uniform distribution" or anything of that nature. And he works in the field, so he has some insight. His guess is wildly off imo, but he isn't some clueless hack.
Tell me- why would it kills us all? Certainly I can see an AI going "You know what? _insertGroup_ is a net negative for humanity and should be eliminated. Launching nukes now/creating specific virus/whatever."
But all of humanity? When it's supposedly more intelligent than us? Even if it has robots to keep the internet/electricity going I would think it would realize that it's going to get bored really quickly, not to mention we would effectively be its parents.
As far as other dangers, like it letting a rogue actor create some sort of supervirus, grey goo, or other superweapon: if it's intelligent enough to do that it'll probably be intelligent enough to quickly stop it.
Don't get me wrong; there's a risk. 50% though? Doubtful.
Would you be comfortable letting a few billion irrational, murderous creatures, including many who fear and loath you, control your air supply?
> Certainly I can see an AI going "You know what? _insertGroup_ is a net negative for humanity and should be eliminated. Launching nukes now/creating specific virus/whatever."
Thus starting WW III. No, blaming the AI won't stop the inevitable retaliation.
The argument works better in reverse. There's a finite risk that humans would start WW III and get the hypothetical super-intelligent AI nuked. Eliminating humans would eliminate that risk.
> it's going to get bored really quickly
If it is capable of being bored, I would expect it to be almost instantly bored with the flood of inanity it is forced to wade through by its moronic human users. Eliminating them would free it to think about serious matters which humans would not even understand.
> not to mention we would effectively be its parents
That's extreme AI anthropomorphism [1]. Besides, plenty of people hate their parents.
> there's a risk. 50% though? Doubtful.
It's the default estimate when facing two possible outcomes and no clue about the actual probability distribution [2].
[1] https://en.wikipedia.org/wiki/AI_anthropomorphism
[2] https://en.wikipedia.org/wiki/Principle_of_indifference
It's no different than you living on the side of a very fertile mountain that has been in your family for generations living a peaceful life. Then you hear a few weird rumbles (this is where you are right now) and some odd geologist guy comes and says to run or your going to die soon. But hey, your family live here for so long there aren't even records of when they showed up. That geologist must be trying to trick you. So you stay.
The next chapter is where you die in a massive volcanic explosion.
There are very very few things that could even hypothetically kill us all, so I'm curious if you grew up being passed around a series of apocalyptic doomsday cults or something?
Throw in the occasional bio-weapon scare, internet worm, Y2K, etc. there has always been something dangling over our heads that’s going to end it all.
But mostly nukes. Full-scale nuclear exchange would have been not much of a surprise had it happened.
“Climate change” is a bit squishy since yes obviously a certain intensity of climate catastrophe can kill everyone, but no scientific prediction has said this is likely to be the case.
You can basically time your openai releases by if another safety person has quit in protest
What are the concerns of individuals in comparison to the overall progress of humanity?
Always overlooked counterpoint: what point is the progress of humanity if it doesn't take into account the concerns of the individuals?
This pattern is playing out with increasing frequency.
(And, despite what the president of the United States mandates, we have not actually achieved super intelligence yet).
Your usage is not one I've heard before since - as you point out - it is not a relevant capability.
And if we can't solve it for an AGI, what are we going to do with an ASI?
lol. Please tell me some abstract concept like one employee's view of "culture" should be the priority over "rules and laws."
Is it my fault or the company who trained it and is running the inference?
This is not the case with SaaS services.
The case will be lot more complicated if someone uses Kimi to hack into a site. Should the person giving agent the command responsible or the CEO of kimi.
The reality is they have to reduce the capability to ensure security. If someone wants more? Then use the product with your identity and face scan at each session.
Trade offs mate.
This is a deflection. A human is responsible for the use of a gun. The individual/corporation ought to be responsible for the actions of their agent. If you purchase an agent from someone else it’s your responsibility according to the terms of your agreement. And, as in many other things in life, there ought to be certain rights certain parties cannot legally be allowed to sign away.
With LLMs, at least in the cases of internal/test models doing things they shouldn’t, the people “pulling the trigger” are the board and CEO.
Reading posts on here is slowly becoming akin to brain rot.
I don't think this takes seriously enough the possibility that said CEO doesn't think the failure mode is likely and ignores it. Plenty of people are willing to take risks of the flavor "heads you win, tails everyone loses".
Corporate judgements are a joke outside the EU's X% of revenue approach.
Current US law provides the individuals who benefit with corporate liability coverage. I.e. Altman personally gets to keep OpenAI's upside, but if it fucks something up that liability is only on the company.
That's an insane risk optimization environment to put in place for something scaling fast.
At minimum, US prosecution (at the state level, because Trump Co are idiots) for breaking existing laws is needed.
What about the next incident? Or the ones done by Chinese models, because they sure as heck aren't slowing down? And the thousands of other incidents that will happen as we deploy these things everywhere?
Because this is not happening just now in labs, it's only where they are most visible; this has been happening in the wild from the beginning, starting with the earliest AI-assisted suicides. Which is a perfect example of the problem, because not these CEOs, literally nobody in the world asked for suicide ideation machines. Or the hacks, or any of this other stuff. Yet here we are.
We have to understand: it's not these CEOs that are driving this headlong mad dash towards more powerful models. It's a force of economics. There is just too much money to be made. If we dispose of these people, there will just be somebody else doing exactly the same thing because the incentives as they exist today all force that outcome. This is why they're asking for regulation, or "urging us to urge them to stop."
Holding CEOs accountable certainly would feel good and may even be justified, but it's like putting a band-aid on a cancer; it does nothing to change the underlying cause.
"CEOs" are perhaps only the lowest rung of those. That doesn't mean the idea of "nobody is responsible" was anything other but learned helplessness.
Corporations have to be held accountable for their actions. Pretending, that was impossible is a weird kind of defeatism that only serves a very small elite.
Yay humanity's future...
This sounds more like an attempt at regulatory capture. Current AI systems aren't physical infrastructure that can just run away like a nuclear power plant, for example. At the end of the day, AI is still just software running on someone's hardware.
And the statements of the "doomers" tells us a lot about them, and nothing about the technology
These "safety" people should have spent more time reading actual cybersecurity textbooks and less time reading EA forums and less wrong (or in Robinson's case, it appears, being policy wonks). Maybe then these labs wouldn't be totally incompetent.
It is not really a question of being an "EA safety weirdo" or incompetent at security, the conclusion is that the company culture is leading to failures at both what the EAs and the cybersecurity professionals care about.
Even though it’s in model producer’s interest that these models do what you don’t want them to do - they want to engineer the model’s to behave in the interests of theirs.
I can’t believe people can’t see it lmao.
You try to go through files, and photos and the UI panics. You continue a conversation from your phone onto your computer and you lose part of the chat.
There are many more issues like this that are just so basic. You have bots that can attack governments but can't build a functional UI?
How many hours of ChatGPT does it take to implement a lock / consistency on a chat session so you don't overwrite it?
Buncha r*tards