LessWrong 2.0 Reader

IYKYK

Do you not see how they could be used here?

This one. I'm confused about what the intuitive intended meaning of the symbol is. Sorry, I see why "type signature" was the wrong way to express that confusion. In my mind a logical counterfactual is a model of the world, with some fact changed, and the consequences of that fact propagated to the rest of the model. Maybe $L_A$ is a boolean fact that is edited? But if so I don't know which fact it is, and I'm confused by the way you described it.

Because we're talking about priors and their influence, all of this is happening inside the agent's brain. The agent is going about daily life, and thinks "hm, maybe there is an evil demon simulating me who will give me -10¹⁰10^10 utility if I don't do what they want for my next action". I don't see why this is obviously ill-defined without further specification of the training setup.

Can we replace this with: "The agent is going about daily life, and its (black box) world model suddenly starts predicting that most available actions actions lead to -10¹⁰ utility."? This is what it's like to be an agent with malign hypotheses in the world model. I think we can remove the additional complication of believing its in a simulation.

I believe the best motivation is compassion without attachment. Being attached to any specific future - we are certain to be doomed or I have this plan that will maybe save us all - is pointless speculation.

Instead, we should use our bayesian inference mind to find the most likely path to help - out of compassion - and pursue it with all our heart. Regardless of whether it works or not, we aren't sad or experiencing suffering since our mind is just filled with compassion. 

If we've done this, then we've done our best.

I’m not saying there are no precise claims about the near future, only that I haven’t made up my mind about those precise claims. For instance, my only active disagreement with the mid-2025 section is that it gives me the impression that LLM agents will be seeing more widespread use than I expect. There are specific claims, like a prediction about SWE bench performance, but I don’t trust SWE bench as a measure of progress towards AGI, and I can’t determine at a glance whether their number is too high or too low.

The later sections are full of predictions that I expect to fail indisputably. The most salient is that AI engineers are supposed to be obselete in like 2 years. 

I don't think it's possible without changing the people into weird types who really don't care too much about the social aspects of life because they're so interested in the topics at hand. You can try rewarding truth, but people still stumble into issues regarding morality, popularity of ideas, the overton window, some political group that they dislike randomly hitting upon the truth so that they look like supporters for stating the same thing, etc.

I think prediction markets are an interesting concept, but it cannot be taken much further than it is now, since the predictions could start influencing the outcomes. It's dangerous to add rewards to the outcomes of predictions, for when enough money is involved, one can influence the outcome.

The way humans in general differ from truth-seeking agents makes their performance downright horrible on some specific areas (if the truth is not in the overton window for instance). These inaccuracies can cascade and cause problems elsewhere, since they cause incorrect worldviews even in somewhat intelligent people like Musk. There's also a lot of information which is simply getting deleted from the internet, and you can't "weight both sides of the argument" if half the argument is only visible on the waybackmachine or archive.md.

I guess it's important to create a good atmosphere and that everyone is having fun theorizing and such, but some of the topics we're discussing are actually serious. The well-being of millions of people depend on the sort of answers and perspectives which float around puvlic discourse, and I find it pathetic that ideas are immediately shut down if they're not worded correctly or if they touch a growing list of socially forbidden hypotheses.

Finally, these alternative rewards have completely destroyed almost all voting systems on the internet. There's almost no website left on which the karma/thumb/upvote/like count bears any resemblence to post quality anymore. Instead, it's a linear combination of superstimuli like 'relatability', 'novelty', 'feeling of importance (e.g. bad news, danger)', 'cuteness', 'escapism', 'sexual fantasy', 'romantic fantasy', 'boo outgroup', 'irony/parody/parody of parody/self-parady/nihilism', 'nostalgia', 'stupidity' (I'm told it's a kind of humor if you're stupid on purpose, but I think "irony" is a defence mechanism against social judgement). It's like a view into the unfulfilled needs of the population. Youtube view count and subscriptions, Reddit karma, Twitter retweets, all almost gamed to the point that they're useless metrics. Online review sites are going in the same direction. It's like interacting with a group of mentally ill people who decide what you're paid each day. I think it's dangerous to upvote comments based on vibes as it takes very little to corrupt these metrics, and it's hard to notice if upvotes gradually come to represent "dopamine released by reading" or something other than quality/truthfulness.

Thanks so much for the comprehensive analysis - this makes it easier to reason about how the political situation and trendlines affect AI development. 

I have some objections that AGI and ASI will be released and that wrapper startups can use them as mentioned below:

Agent-3-mini is hugely useful for both remote work jobs and leisure. An explosion of new apps and B2B SAAS products rocks the market. Gamers get amazing dialogue with lifelike characters in polished video games that took only a month to make. 10% of Americans, mostly young people, consider an AI “a close friend.” For almost every white-collar profession, there are now multiple credible startups promising to “disrupt” it with AI.

1. Once this AGI is there, it can recreate all these B2B SaaS at a cheap cost since SaaS is mostly plug and play / doesn't depend on specific company IP / resources
2. B2B SaaS created by OpenBrain likely have more credibility than most other B2B SaaS as OpenBrain likely has a large reputation and following as the leading AI company
3. If OpenBrain owns all B2B SaaS, their profits, resources, and compute likely increase
4. An API to AGI will allow competitors to develop their own AGI by accelerating their foundation model development
5. From competitive pressures and more profit, AGI will not be released and instead used to create services.

I am also much more pessimistic about the endings after September 2027 on the website for the following reasons:

1. It took humans to help detect and rollback misaligned AI in the predictions, which will likely happen
2. There is likely no complete alignment in the AI, given specific context that is out of training distribution, it will likely scheme, even if it's 0.0001% of the time
3. These misalignment issues will likely accumulate as it progressively trains strong superintelligence
4. Humans will not be able to tell at all what stronger superintelligence are doing - they could only barely tell with the weaker superintelligence
5. Finally, a sufficiently misaligned superintelligence replaces all humans with new lifeforms as in the pessimistic ending to satisfy its value function

I can accept the argument that even if AI are misaligned 0.0001% of the time, this misalignment will only decrease over time as AI can generalize better, but I find this too optimistic.

Thanks again for working on this, hopefully this will inform government policy in the right direction.

If a strategy is likely to be outdated quickly it's not robust and not a good strategy. Strategies should be able to withstand lots of variation.

I'm not sure what the type signature of $L_A$ is, or what it means to "not take into account $M$'s simulation"

I know you know about logical decision theory, and I know you know its not formalized, and I'm not going to be able to formalize it in a LessWrong comment, so I'm not sure what you want me to say here. Do you reject the idea of logical counterfactuals? Do you not see how they could be used here?

I think you've misunderstood me entirely. Usually in a decision problem, we assume the agent has a perfectly true world model, and we assume that it's in a particular situation (e.g. with omega and knowing how omega will react to different actions). But in reality, an agent has to learn which kind of world its in using an inductor. That's all I meant by "get its beliefs".

Because we're talking about priors and their influence, all of this is happening inside the agent's brain. The agent is going about daily life, and thinks "hm, maybe there is an evil demon simulating me who will give me -10¹⁰10^10 utility if I don't do what they want for my next action". I don't see why this is obviously ill-defined without further specification of the training setup.

Can you give something specific? It seems like pretty much every statement has a footnote grounding the relevant high-level claim in low-level indicators, and in cases where that's not the case, those predictions often seem clear derivatives of precise claims in eg their compute forecast

I'm not sure what the type signature of $L_A$ is, or what it means to "not take into account $M$'s simulation". When $A$ makes decisions about which actions to take, it doesn't have the option of ignoring the predictions of its own world model. It has to trust its own world model, right? So what does it mean to "not take it into account"?

So the way in which the agent "gets its beliefs" about the structure of the decision theory problem is via these logical-counterfactual-conditional operation

I think you've misunderstood me entirely. Usually in a decision problem, we assume the agent has a perfectly true world model, and we assume that it's in a particular situation (e.g. with omega and knowing how omega will react to different actions). But in reality, an agent has to learn which kind of world its in using an inductor. That's all I meant by "get its beliefs".