LessWrong 2.0 Reader

Essentially you seem to want more of the same of what we had for the past decades: more cheap goods and loss of production know-how and all that goes along with it. This feels a bit funny as (i) just in the recent years many economists, after having been dead-sure that old pattern would only mean great benefits, may not quite be so cool overall (covid exposing risky dependencies, geopolitical power loss, jobs...), and (ii) your strongman in power shows to what it leads if we only think of 'surplus' (even your definition) instead of things people actually care about more (equality, jobs, social security..).

You'd still be partly right if the world was so simple that handing the trade partners your dollars would just mean we reprint more of it. But instead, handing them your dollars gives them global power; leverage over all the remaining countries in the world, as they have now the capability to produce everything cheaply for any other country globally, plus your dollars to spend on whatever they like in the global marketplace for products and influence over anyone. In reality, your imagined free lunch isn't quite so free.

So gotta keep in mind that probabilities are in your head (I flip a coin, it's already tails or heads in reality, but your credence should still be 50-50). I think it can be the case that we were always doomed even if weren't yet justified in believing that.

Alternatively, it feels like this pushes up against philosophies of determinism and freewill. The whole "well the algorithm is a written program and it'll choose what is chooses deterministically" but also from the inside there are choices.

I think a reason to have been uncertain before and update more now is just that timelines seem short. I used to have more hope because I thought we had a lot more time to solve both technical and coordination problems, and then there was the DL/transformers surprise. You make a good case and maybe 50 years more wouldn't make a difference, but I don't know, I wouldn't have as high p-doom if we had that long.

That does not look like state valued consequentialism as we typically see it, but as act valued consequentialism (In markov model this is sum of value of act (intrinsic), plus expected value of the sum of future actions) action agent with value on the acts, Use existing X to get more Y and Use existing Y to get more X.  I mean, how is this different from the value on the actions X producing y, and actions Y producing X, if x and Y are scale in a particular action. 

It looks money pump resistant because it wants to take those actions as many times as possible, as well as possible, and a money pump generally requires that the scale of the transactions drops over time (the resources the pumper is extracting). But then the trade is inefficient. There is probably benefits for being an efficient counterparty, but moneypumpers are inefficient counterparties. 

I haven't read most of the paper, but based on the Extended Abstract I'm quite happy about both the content and how DeepMind (or at least its safety team) is articulating an "anytime" (i.e., possible to implement quickly) plan for addressing misuse and misalignment risks. But I think safety at Google DeepMind is more bottlenecked by buy-in from leadership to do moderately costly things than the safety team having good plans and doing good work.

I suspect that, to many readers, what gives urgency to the Krome claims is that two people have allegedly died  at the facility. For example, the fourth link OP provides is an instagram video with the caption “people are dying under ICE detainment in Miami”.

The two deceased are Genry Ruiz Guillen and Maksym Chernyak. ICE has published deaths reports for both:

https://www.ice.gov/doclib/foia/reports/ddr-GenryRuizGuillen.pdf

https://www.ice.gov/doclib/foia/reports/ddrMaksymChernyak.pdf

Notably, Mr. Ruiz-Guillen was transferred to medical and psychiatric facilities multiple times, and my read of the timeline is that he was in the custody of various hospitals from December 11 up through his January 23 death, i.e. over a month separates his death and his time at Krome. (It’s possible I’m reading this wrong so let me know if others have a different read). Ruiz-Guillen was transferred to hospital a month before inauguration day.

Chernyak’s report is much shorter and I don’t know what to make of it. Hemmorhagic stroke is hypothesized. He died February 20.

These are fairly detailed timelines. Guillen-Ruiz’s in particular involves many parties (normal hospital, psychiatric hospital, different doctors), so would be a pretty bold fabrication.

You said:

>the fact that we haven't seen definitive evidence against the allegations is significant evidence in favour of their veracity.

But “detainees are dying because of overcrowding and lack of water” is an allegation made by one of OP’s links, and these timelines and symptoms, especially Guillen-Ruiz’s, are evidence against.

I personally first deeply felt the sense of "I'm doomed, I'm going to die soon" almost exactly a year ago, due to a mix of illness and AI news. It was a double-whammy of getting both my mortality, and AGI doom, for the very first time.

Re: mortality, it felt like I'd been immortal up to that point, or more accurately a-mortal or non-mortal or something. Up to this point I hadn't anticipated death happening to me as anything more than a theoretical exercise. I was 35, felt reasonably healthy, was familiar with transhumanism, had barely witnessed any deaths in the family, etc. I didn't feel like a mortal being that can die very easily, but more like some permanently existing observer watching a livestream of my life: it's easy to imagine turning off the livestream, but much harder to imagine that I, the observer, will eventually turn off.

After I felt like I'd suddenly become mortal, I experienced panic attacks for months.

Re: AGI doom: even though I've thought way less about this topic than you, I do want to challenge this part:

And doomed for no better reason than because people were incapable of not doing something.

Just as I felt non-mortal because of an anticipated transhumanist future or something, so too did it feel like the world was not doomed, until one day it was. But did the probability of doom suddenly jump to >99% in the last few years, or was the doom always the default outcome and we were just wrong to expect anything else? Was our glorious transhumanist future taken from us, or was it merely a fantasy, and the default outcome was always technological extinction?

Are we in a timeline where a few actions by key players doomed us, or was near-term doom always the default overdetermined outcome? Suppose we go back to the founding of LessWrong in 2009, or the founding of OpenAI in 2015. Would a simple change, like OpenAI not being founded, actually meaningfully change the certainty of doom, or would it have only affected the timeline by a few years? (That said, I should stress that I don't absolve anyone who dooms us in this timeline from their responsibility.)

From my standpoint now in 2025, AGI doom seems overdetermined for a number of reasons, like:

• Humans are the first species barely smart enough to take over the planet, and to industrialize, and to climb the tech ladder. We don't have dath ilan's average IQ of 180 or whatever. The arguments for AGI doom are just too complicated and time-consuming to follow, for most people. And even when people follow them, they often disagree about them.
• Our institutions and systems of governance have always been incapable of fully solving mundane problems, let alone extinction-level ones. And they're best at solving problems and disasters they can actually witness and learn from, which doesn't happen with extinction-level problems. And even when we faced a global disaster like Covid-19, our institutions didn't take anywhere sufficient steps to prevent future pandemics.
• Capitalism: our best, or even ~only, truly working coordination mechanism for deciding what the world should work on is capitalism and money, which allocates resources towards the most productive uses. This incentivizes growth and technological progress. There's no corresponding coordination mechanism for good political outcomes, incl. for preventing extinction.
• In a world where technological extinction is possible, tons of our virtues become vices:
  • Freedom: we appreciate freedoms like economic freedom, political freedom, and intellectual freedom. But that also means freedom to (economically, politically, scientifically) contribute to technological extinction. Like, I would not want to live in a global tyranny, but I can at least imagine how a global tyranny could in principle prevent AGI doom, namely by severely and globally restricting many freedoms. (Conversely, without these freedoms, maybe the tyrant wouldn't learn about technological extinction in the first place.)
  • Democracy: politicians care about what the voters care about. But to avert extinction you need to make that a top priority, ideally priority number 1, which it can never be: no voter has ever gone extinct, so why should they care?
  • Egalitarianism: resulted in IQ denialism; if discourse around intelligence was less insane, that would help discussion of superintelligence.
  • Cosmopolitanism: resulted in pro-immigration and pro-asylum policy, which in turn precipitated both a global anti-immigration and an anti-elite backlash.
  • Economic growth: the more the better; results in rising living standards and makes people healthier and happier... right until the point of technological extinction.
  • Technological progress: I've used a computer, and played video games, all my life. So I cheered for faster tech, faster CPUs, faster GPUs. Now the GPUs that powered my games instead speed us up towards technological extinction. Oops.
• And so on.

Yudkowsky had a glowfic story about how dath ilan prevents AGI doom, and that requires a whole bunch of things to fundamentally diverge from our world. Like a much smaller population; an average IQ beyond genius-level; fantastically competent institutions; a world government; a global conspiracy to slow down compute progress; a global conspiracy to work on AI alignment; etc.

I can imagine such a world to not blow itself up. But even if you could've slightly tweaked our starting conditions from a few years or decades ago, weren't we going to blow ourselves up anyway?

And if doom is sufficiently overdetermined, then the future we grieve for, transhumanist or otherwise, was only ever a mirage.

You are right. 

I had an interesting conversation about this.

This is in contrast to consequentialism as a part/tool of other moral systems

If there is a deotological rule don't murder, then the question becomes, what makes an action murder.

Murder is when you kill someone. (Obviously). 

That is, murder is an action with the consequence of death (to rounding)

So there is a sense in which consequentialism is also for saying which actions fall into which categories in some other moral framework. 

I retract. 

"What actions are what" is a question only a consequentialist would ask (consequentialism grows out of the ontology of figuring out the consequences of actions). 

Other moral systems can/do exist in ontologies where you do not believe that is possible, or do not trust that it is possible to be good at that, and so judge on other grounds. 

Downvoted, the post contained a bobcat.

(Not really)

By the way, I don't mean to imply that Meta AI doesn't have talented AI researchers working there. The problem is more that the competent minority are so diluted and hampered by bureaucratic parasites that they can't do their jobs properly.

...EN argues that any sufficiently expressive cognitive system—such as the human brain—must generate internal propositions that are arithmetically undecidable. These undecidable structures function as evolutionarily advantageous analogues to Gödel sentences, inverted into the belief in raw subjective experience (qualia), despite being formally unprovable within the system itself.

Rather than explaining subjective illusions away in third-person terms, EN proposes that they arise as formal consequences of self-referential modeling, constrained by the expressive limits of second-order logic.

How could undecidability, unprovability, self-referential modeling, incompleteness, or any sort of logic generate the redness of red?

Incomplete, self-referential modeling → ? → red

The brain does this by creating a symbol, which refers to a symbol, which refers to a symbol—an infinite regress with no grounding, no bottom. But evolution doesn’t need grounding; it needs action. So it skewed this looping process toward stability—toward a fixed point. That fixed point is the assertion: “I exist.” Not because the system proves it, but because the loop collapses into a self-reinforcing structure that feels true. This is not the discovery of a self—it’s the compression artifact of a system trying to model itself through unprovable means. The result is a symbol that mistakes itself for a subject.

The feeling part remains unexplained.

What justifies a formal system becoming experience?

Well, that’s the heart of the matter: ultimately, nothing.

Experience is not something we have, but something we enact. Your experience is barred from being “real” in any ontologically grounded sense because the universe cannot produce something like it directly. Yet it can still be consistent, much like a force—both can only be inferred from their effects.

We still seem to have experience. How can this "seeming" feel like something? If you boil everything down to math, how can math feel like anything?

One: Qualia are not illusions, they are fictions.

Why do people defend qualia so intensely if they’re illusions?

Because the illusion is evolutionarily entrenched and cognitively reinforced.

This seems contradictory.

A defendant guilty of homicide argues that, due to EN, the victim had no conscious experience and thus suffered no moral harm. The judge, also an EN advocate, counters that if consciousness is illusory, the defendant’s claim of injustice itself collapses. Ethical responsibility remains intact irrespective of qualia's ontological status.

What about torturing animals?